Saving GPX Files to Azure Blob Storage

Recap

In the last post we gathered the track data from Google Maps, feed it into GPS Babel and have our GPX formatted file saved to a local directory. Now we want to enable our users to put a name to the file and store it in our blob storage account for later use.

GPX Viewer

We need a way for the user to validate the output from the conversion process and specify a name for the track. We can fulfill both requirements with a single form:

A text box to display the file GPX data, one for the file name and a save button are all we need. Create a public property to hold the GPX data as a string. In the form's shown event, assign it to the large text box. One more detail, GPS Babel output the GPX in the GPX/1/0 format and we want the GPX/1/1. A simple fix is to replace the xmlns property like so:

private void Document_Shown(object sender, EventArgs e)
        {
            Doc = Doc.Replace("http://www.topografix.com/GPX/1/0", "http://www.topografix.com/GPX/1/1");

            editDocument.Text = Doc;
        }

Save That Track!

Once the user enters a file name and presses the Save button we have two tasks; first validate the file name and then save the file to our Azure storage account.

Blob file names follow the same rules as windows file names, so we can use the built in .net function (GetInvalidFileNameChars) for this:

private bool FileNameIsValid()
        {
            if (string.IsNullOrEmpty(txtFileName.Text))
            {
                MessageBox.Show(
                    "Please enter a valid file name for this track.",
                    "No File Name",
                    MessageBoxButtons.OK,
                    MessageBoxIcon.Error);

                return false;
            }

            if (txtFileName.Text.IndexOfAny(System.IO.Path.GetInvalidFileNameChars()) != -1)
            {
                MessageBox.Show(
                    "Please enter a valid file name for this track.",
                    "Invalid File Name",
                    MessageBoxButtons.OK,
                    MessageBoxIcon.Error);

                return false;
            } 

            return true;
        }

An Aside About Blobs

There are two kinds of blobs: block and page.

Block blobs allow a single blob to be broken up into smaller blocks. These blocks allow parallel upload/download thus allowing for better performance. They are limited to 200GB in size. Each block can be up to 4MB in size (allowing for 50,000 blocks). Each block must be uploaded and then the entire blob is committed into storage. That means uploading block blobs is a two-step process. You can upload the blob in a single operation when the block blob is less than 64MB.

A page blob is collection of pages. Individual pages can be up to 1 TB, but each page must be a multiple of 512 bytes. A page is a range of data that is identified by its offset from the start of the blob. Pages can be randomly uploaded and accessed. Unlike block blobs, writes to a page blob are committed immediately.

Which should you use and when?

Well that depends on your file size and your usage scenario. If you have no need to pull individual pages and your files are < 200GB then use block blobs. Otherwise you will need to use page blobs. For our purposes block will do just fine.

Now Back To Our Regularly Scheduled Post

Wire up the Save button like so:

private void cmdSave_Click(object sender, EventArgs e)
        {
            if (FileNameIsValid())
            {
                var containerName = ConfigurationManager.AppSettings["DemoTrackContainer"];
                var fileName = txtFileName.Text + ".gpx";

                if (Blob.CreateBlockBlob(containerName, fileName, editDocument.Text))
                {
                    MessageBox.Show(
                        fileName + " was sucessfully saved.",
                        "Demo Track Saved",
                        MessageBoxButtons.OK,
                        MessageBoxIcon.Information);

                    Close();
                }
                else
                {
                    MessageBox.Show(
                        "There was an error saving " + fileName + ".",
                        "Error Saving Demo Track",
                        MessageBoxButtons.OK,
                        MessageBoxIcon.Error);
                }
            }
        }

Here you can see that we test our file name, grab our container name from app settings and add the ".gpx" extension to our file name. We then call a static class called Blob and it's CreateBlockBlob method. The method creates a file in the specified container with the passed string as it content. Here is the code:

public static bool CreateBlockBlob(string containerName, string fileName, string text)
        {
            try
            {
                var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
                CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

                CloudBlobContainer container = blobClient.GetContainerReference(containerName);
                container.CreateIfNotExist();

                CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
                blob.UploadText(text);

                return true;
            }
            catch (Exception)
            {
                return false;
            }
        }

And here is the code for creating a page blob (in case you were curious)

public static bool CreatePageBlob(string containerName, string fileName, string text, long size)
        {
            try
            {
                var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
                CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

                CloudBlobContainer container = blobClient.GetContainerReference(containerName);
                container.CreateIfNotExist();

                CloudPageBlob blob = container.GetPageBlobReference(fileName);
                blob.Create(size);
                blob.UploadText(text);

                return true;
            }
            catch (Exception)
            {
                return false;
            }
        }

Now we are saving our tracks in the cloud (yea for marketing slang!) and have then available for future use. Speaking of the future, the next post we will get started on the track faker piece of the application. We are going to take advantage of Tasks to handle multi-threading so we will continue to have a responsive UI while faking a long running track.

Generating Fake GPS Data

Why Fake Data?

One of the more interesting bits to the project is being able to generate fake GPS tracks to feed to the API. We need to be able to demonstrate the real-time aspects of the solution without having to have someone drive around and collect the data. Also it would be nice be able to generate tracks based on who we are demoing to. Perhaps a simulated drive from the iVision office to a clients office.

So, the goal is to be able to generate fake GPS tracks that we can feed to the iGOR application while demoing the real-time updating of the maps. We also need to be able to generate these tracks from any two points, and they should follow the streets and roads available in a logical way.

Google driving directions seems like a good place to start, and I found a great tool to help. GPS Babel is a free application that can take the data generated from Google Maps (and many other sources) and generate a standard GPX file.

So we want a page in our application that looks something like this:

We will have the user enter their start and destination addresses into Google Maps hosted in a browser control. They will then copy the link generated from the directions into the text box and press the Go button.

This will save the resulting HTML to a file that can be picked up by GPS Babel and converted into GPX. A lot more straight forward than it sounds, I promise.

First thing to do is download/install GPS Babel (follow the link above). Make a note of the location you install it as you will need it later. Now in our web browser control we can set our url to http://map.google.com.

Now lets wire up our Go button event like this:

private void cmdGo_Click(object sender, EventArgs e)
        {
            try
            {
                _isBuilding = true;
                var newURI = new Uri(txtTrackURI.Text + "&output=js");
                webBrowser.ScriptErrorsSuppressed = true;
                webBrowser.Url = newURI;
            }
            catch (Exception)
            {
                MessageBox.Show("Link does not appear to be valid. Please chech the track uri and try again.",
                                "Invalid Track URI", MessageBoxButtons.OK, MessageBoxIcon.Asterisk,
                                MessageBoxDefaultButton.Button1);
                _isBuilding = false;
            }
        }

As you can see we set a form level bool to true (I will come back to this later), take the text from the textbox and append a query parameter. This parameter tells Google to output the resulting page in a special format that can be imported by GPS Babel. We then update the web browser controls uri to this new location. Lets take a look at the Document Complete event of our control:

        private void webBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (_isBuilding)
            {
                var gpsBabelLocation = Properties.Settings.Default.GPSBabelLocation + @"\gpsbabel.exe";
                var tempFolder = Properties.Settings.Default.LocalTempFolder;

                var localMapFilePath = tempFolder + @"\map.txt";
                var localGPXFilePath = tempFolder + @"\gpx_data.gpx";

                var fs = new FileStream(localMapFilePath, FileMode.Create, FileAccess.Write, FileShare.None);
                var file = new StreamWriter(fs, Encoding.ASCII);
                file.WriteLine(webBrowser.DocumentText);
                file.Close();

                var p = new Process
                {
                    StartInfo =
                    {
                        FileName = gpsBabelLocation,
                        Arguments = @"-t -i google -f " + localMapFilePath + @" -o gpx,suppresswhite=0,logpoint=0,humminbirdextensions=0,garminextensions=0 -F " + localGPXFilePath,
                        UseShellExecute = false
                    }
                };

                p.Start();
                p.WaitForExit();

                TextReader tr = new StreamReader(localGPXFilePath);
                var viewer = new GPXViewer { Doc = tr.ReadToEnd() };
                tr.Close();
                viewer.ShowDialog();

                webBrowser.GoBack();
                txtTrackURI.Text = null;
            }

            _isBuilding = false;
        }

Ah, there is that _isBuilding again. As you can see we are using it to control whether or not we process the web controls contents on document complete. As the user is generating their track, this event gets fired several times and we only want to process it when the user has hit the Go button.

The first thing we do is load up the location of the GPS Babel exe and the location of a temp folder (these are stored via the Settings tab which I will cover in a different post). We also set up the location of the map.txt file which is the input into GPS Babel, and gpx_data.gpx which is the output from the conversion process.

Next we save the contents of the web browser control to our input file (map.txt). Now we create a new Process object to kick off the GPS Babel exe. We give it the exe location and a set of command line arguments to control how GPS Babel executes and where to store the output. Once the conversion process is complete we read in the results from the gpx_data.gpx file into a string and pass it to another form to display the output.

My next post will cover the gpx viewer form and saving our file to Azure cloud storage for later retrieval.