Showing posts with label Azure. Show all posts
Showing posts with label Azure. Show all posts

Wednesday, August 20, 2014

Orchestrate Provider for Project Orleans

Previously we discussed Orleans as a new middle tier, actor model framework from Microsoft. Today I want to discuss creating a custom persistence provider using Orchestrate.io as the storage mechanism.

The Orleans Persistence Model

Persistence in Orleans is a simple declarative model where you identify the data to be saved in permanent storage via convention, and the programmer controls when and where the data is stored. Using this model is not required however, you can roll your own. 

How It Works

You declare what data needs to be saved using the IGrainState interface, and you pass this interface into the GrainBase when creating your grain class. You will also need to set a reference to the provider in the host project and set the provider type in the server configuration XML. Once this is done, the framework will attempt to load the grain's state information from permanent storage on activation. Saving is up to the developer and is done with a simple call to the provider's WriteStateAsync() method.

The Provider Interface

Orleans provides an IStorageProvider interface that we must implement if we are going to create an Orchestrate provider. It is fairly simple fortunately and here it is:

class OrchestrateProvider : IStorageProvider
{
    #region IStorageProvider Members

    public Task ClearStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.GrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task Close()
    {
        throw new NotImplementedException();
    }

    public Orleans.OrleansLogger Log
    {
        get { throw new NotImplementedException(); }
    }

    public Task ReadStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task WriteStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    #endregion

    #region IOrleansProvider Members

    public Task Init(string name, Orleans.Providers.IProviderRuntime providerRuntime, Orleans.Providers.IProviderConfiguration config)
    {
        throw new NotImplementedException();
    }

    public string Name
    {
        get { throw new NotImplementedException(); }
    }

#endregion
}

Orchestrate Provider

Lets start with the OrleansProvider members. These are bits that set up the storage mechanism on first use. The name property interface can be satisfied with a simple private string with a public getter. I will leave that to you to do on your own. The Init task is a bit more interesting. Here we need to instantiate an Orchestrate.NET instance and configure it with our API key.

In the init function we will set the name property, get our API key and instantiate our Orchestrate.NET instance. The config parameter has a dictionary of all of the items declared in the server configuration file.

public Task Init(string name, IProviderRuntime providerRuntime, IProviderConfiguration config)
{
    Name = name;

    if (string.IsNullOrWhiteSpace(config.Properties["APIKey"])) 
        throw new ArgumentException("APIKey property not set");

    var apiKey = config.Properties["APIKey"];

    _orchestrate = new Orchestrate.Net.Orchestrate(apiKey);

    return TaskDone.Done;
}

Next lets set up our ReadStateAsync. Now we have a decision to make, what will be the name of our Orchestrate collection and what will be the items key? We will use the grain state's type name for the name of our collection and the grains key for our key. As you know when we read items out of Orchestrate we will get back a json string. So the next step is to deserialize it back out to the grain state type and cast that to IGrainState. Now we can call the grain states SetAll() method and we are set. Let's see the code:

public async Task ReadStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    try
    {
        var results = await _orchestrate.GetAsync(collectionName, key);

        var dict =     ((IGrainState)JsonConvert.DeserializeObject(results.Value.ToString(),       grainState.GetType())).AsDictionary();
        grainState.SetAll(dict);
    }
    catch (Exception ex)
    {
        Console.WriteLine("==> No record found in {0} collection for id {1}\n\r", new object[] { collectionName, key });
        WriteStateAsync(grainType, grainReference, grainState);
    }
}

Now if we have an exception thrown, it means that item does not exist in our collection. Because all grains always exist in Orleans, we will go ahead and write the item to our collection.

Speaking of writing, lets go ahead and look at the WriteStateAsync code:

public async Task WriteStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    try
    {
        var results = await _orchestrate.PutAsync(collectionName, key, grainState);
    }
    catch (Exception ex)
    {
        Console.WriteLine("==> Write failed in {0} collection for id {1}", new object[] { collectionName, key });
    }
}

And the ClearStateAsync:

public async Task ClearStateAsync(string grainType, GrainReference grainReference, GrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    await _orchestrate.DeleteAsync(collectionName, key, false);
}

The last method is the Close, with the Orchestrate.NET provider we can simply set our instance to null and be done. I will leave that code to you.

Wrap Up

And that is all there is to it. Check out the source code and sample project on github. I believe Orleans will have a place in your tool box and Orchestrate makes for a powerful persistence mechanism to pair with it.

If you are going to download and run the code yourself, make sure you create an app in Orchestrate, create ManagerState and EmployeeState collections, then grab your API key and put it at the appropriate place in the DevTestServerConfiguration.xml file.


Thursday, August 14, 2014

Project "Orleans" and Orchestrate.NET

Cloud applications are by default distributed and require parallel processing. Modern application users demand near real-time interaction and responses. Project "Orleans" is Microsoft's new framework for creating cloud (Azure) based distributed applications that meet these requirements. Based on established .Net code and practices, it brings the actor model to your toolbox.

Orleans

Orleans is a new middle tier framework that allows you to "cache" your business objects and their data. It accomplishes this with "grains". A grain is a single-threaded, encapsulated, light weight object that can communicate with other grains via asynchronous message passing. Grains are hosted in "Silos", typically one silo per server. Each silo in an application knows of the other silos and can pass grains and messages between them. Two main goals of Orleans is developer productivity and scaleability by default.

Developer Productivity

Productivity is achieved by providing a familiar environment for development. Grains are .Net objects with declared interfaces, this allows the grains to appear as simple remote objects that can be interacted with directly. Grains are also guaranteed to be single threaded, therefore the programmer never has to deal with locks or other synchronization methods to control access to shared resources. Grains are activated as needed, and if not in use can be garbage collected transparently. This makes grains behave like they are in a cache, "paged-in"/"paged-out" as required. The location of grains is transparent to the developer as well, programmers never need be concerned about which silo a grain is in, as all messaging is handled by the framework.

Scaleability

Fine-grain grains? Orleans makes it easy to break middle tier objects into small units. It can handle large numbers of actors with ease, millions or more, this allows Orleans to control what grains are active and where. Heavy loads on a specific section of grains will be load balanced automatically by the Orleans framework. Grains have logical end-points, with messaging multiplexed across a set of all-to-all physical connections via TCP, this allows a large number of addressable grains with low OS overhead. The Orleans run-time can schedule a large number of grains across a custom thread pool, allowing the framework to run at a high CPU utilization rate with high stability. The nature of messaging between actors allows programmers to develop non-blocking asynchronous code, allowing for a higher degree of parallelism and throughput without using multi-threading in the grains themselves.

Orchestrate.NET

Where does persistence come in? Orleans allows programmers to persist grain state via an integrated store. The grains will synchronize updates and guarantee that callers receive results only after the state has been successfully saved. This system is easily customized/extended and we will do so, and use Orchestrate.NET as the storage provider.

But not until the next post... Meantime you can read up on Orleans, Orleans @ build, Orchestrate and Orchestrate.NET.

Monday, June 4, 2012

Saving GPX Files to Azure Blob Storage

Recap

In the last post we gathered the track data from Google Maps, feed it into GPS Babel and have our GPX formatted file saved to a local directory. Now we want to enable our users to put a name to the file and store it in our blob storage account for later use.

GPX Viewer

We need a way for the user to validate the output from the conversion process and specify a name for the track. We can fulfill both requirements with a single form:


A text box to display the file GPX data, one for the file name and a save button are all we need. Create a public property to hold the GPX data as a string. In the form's shown event, assign it to the large text box. One more detail, GPS Babel output the GPX in the GPX/1/0 format and we want the GPX/1/1. A simple fix is to replace the xmlns property like so:

private void Document_Shown(object sender, EventArgs e)
        {
            Doc = Doc.Replace("http://www.topografix.com/GPX/1/0", "http://www.topografix.com/GPX/1/1");

            editDocument.Text = Doc;
        }

Save That Track!

Once the user enters a file name and presses the Save button we have two tasks; first validate the file name and then save the file to our Azure storage account.

Blob file names follow the same rules as windows file names, so we can use the built in .net function (GetInvalidFileNameChars) for this:

private bool FileNameIsValid()
        {
            if (string.IsNullOrEmpty(txtFileName.Text))
            {
                MessageBox.Show(
                    "Please enter a valid file name for this track.",
                    "No File Name",
                    MessageBoxButtons.OK,
                    MessageBoxIcon.Error);

                return false;
            }

            if (txtFileName.Text.IndexOfAny(System.IO.Path.GetInvalidFileNameChars()) != -1)
            {
                MessageBox.Show(
                    "Please enter a valid file name for this track.",
                    "Invalid File Name",
                    MessageBoxButtons.OK,
                    MessageBoxIcon.Error);

                return false;
            } 

            return true;
        }

An Aside About Blobs

There are two kinds of blobs: block and page.

Block blobs allow a single blob to be broken up into smaller blocks. These blocks allow parallel upload/download thus allowing for better performance. They are limited to 200GB in size. Each block can be up to 4MB in size (allowing for 50,000 blocks). Each block must be uploaded and then the entire blob is committed into storage. That means uploading block blobs is a two-step process. You can upload the blob in a single operation when the block blob is less than 64MB.

A page blob is collection of pages. Individual pages can be up to 1 TB, but each page must be a multiple of 512 bytes. A page is a range of data that is identified by its offset from the start of the blob. Pages can be randomly uploaded and accessed. Unlike block blobs, writes to a page blob are committed immediately.

Which should you use and when?

Well that depends on your file size and your usage scenario. If you have no need to pull individual pages and your files are < 200GB then use block blobs. Otherwise you will need to use page blobs. For our purposes block will do just fine.

Now Back To Our Regularly Scheduled Post

Wire up the Save button like so:

private void cmdSave_Click(object sender, EventArgs e)
        {
            if (FileNameIsValid())
            {
                var containerName = ConfigurationManager.AppSettings["DemoTrackContainer"];
                var fileName = txtFileName.Text + ".gpx";

                if (Blob.CreateBlockBlob(containerName, fileName, editDocument.Text))
                {
                    MessageBox.Show(
                        fileName + " was sucessfully saved.",
                        "Demo Track Saved",
                        MessageBoxButtons.OK,
                        MessageBoxIcon.Information);

                    Close();
                }
                else
                {
                    MessageBox.Show(
                        "There was an error saving " + fileName + ".",
                        "Error Saving Demo Track",
                        MessageBoxButtons.OK,
                        MessageBoxIcon.Error);
                }
            }
        }

Here you can see that we test our file name, grab our container name from app settings and add the ".gpx" extension to our file name. We then call a static class called Blob and it's CreateBlockBlob method. The method creates a file in the specified container with the passed string as it content. Here is the code:

public static bool CreateBlockBlob(string containerName, string fileName, string text)
        {
            try
            {
                var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
                CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

                CloudBlobContainer container = blobClient.GetContainerReference(containerName);
                container.CreateIfNotExist();

                CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
                blob.UploadText(text);

                return true;
            }
            catch (Exception)
            {
                return false;
            }
        }


And here is the code for creating a page blob (in case you were curious)

public static bool CreatePageBlob(string containerName, string fileName, string text, long size)
        {
            try
            {
                var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
                CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

                CloudBlobContainer container = blobClient.GetContainerReference(containerName);
                container.CreateIfNotExist();

                CloudPageBlob blob = container.GetPageBlobReference(fileName);
                blob.Create(size);
                blob.UploadText(text);

                return true;
            }
            catch (Exception)
            {
                return false;
            }
        }

Now we are saving our tracks in the cloud (yea for marketing slang!) and have then available for future use. Speaking of the future, the next post we will get started on the track faker piece of the application. We are going to take advantage of Tasks to handle multi-threading so we will continue to have a responsive UI while faking a long running track.