Wednesday, August 20, 2014

Orchestrate Provider for Project Orleans

Previously we discussed Orleans as a new middle tier, actor model framework from Microsoft. Today I want to discuss creating a custom persistence provider using Orchestrate.io as the storage mechanism.

The Orleans Persistence Model

Persistence in Orleans is a simple declarative model where you identify the data to be saved in permanent storage via convention, and the programmer controls when and where the data is stored. Using this model is not required however, you can roll your own. 

How It Works

You declare what data needs to be saved using the IGrainState interface, and you pass this interface into the GrainBase when creating your grain class. You will also need to set a reference to the provider in the host project and set the provider type in the server configuration XML. Once this is done, the framework will attempt to load the grain's state information from permanent storage on activation. Saving is up to the developer and is done with a simple call to the provider's WriteStateAsync() method.

The Provider Interface

Orleans provides an IStorageProvider interface that we must implement if we are going to create an Orchestrate provider. It is fairly simple fortunately and here it is:

class OrchestrateProvider : IStorageProvider
{
    #region IStorageProvider Members

    public Task ClearStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.GrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task Close()
    {
        throw new NotImplementedException();
    }

    public Orleans.OrleansLogger Log
    {
        get { throw new NotImplementedException(); }
    }

    public Task ReadStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    public Task WriteStateAsync(string grainType, Orleans.GrainReference grainReference, Orleans.IGrainState grainState)
    {
        throw new NotImplementedException();
    }

    #endregion

    #region IOrleansProvider Members

    public Task Init(string name, Orleans.Providers.IProviderRuntime providerRuntime, Orleans.Providers.IProviderConfiguration config)
    {
        throw new NotImplementedException();
    }

    public string Name
    {
        get { throw new NotImplementedException(); }
    }

#endregion
}

Orchestrate Provider

Lets start with the OrleansProvider members. These are bits that set up the storage mechanism on first use. The name property interface can be satisfied with a simple private string with a public getter. I will leave that to you to do on your own. The Init task is a bit more interesting. Here we need to instantiate an Orchestrate.NET instance and configure it with our API key.

In the init function we will set the name property, get our API key and instantiate our Orchestrate.NET instance. The config parameter has a dictionary of all of the items declared in the server configuration file.

public Task Init(string name, IProviderRuntime providerRuntime, IProviderConfiguration config)
{
    Name = name;

    if (string.IsNullOrWhiteSpace(config.Properties["APIKey"])) 
        throw new ArgumentException("APIKey property not set");

    var apiKey = config.Properties["APIKey"];

    _orchestrate = new Orchestrate.Net.Orchestrate(apiKey);

    return TaskDone.Done;
}

Next lets set up our ReadStateAsync. Now we have a decision to make, what will be the name of our Orchestrate collection and what will be the items key? We will use the grain state's type name for the name of our collection and the grains key for our key. As you know when we read items out of Orchestrate we will get back a json string. So the next step is to deserialize it back out to the grain state type and cast that to IGrainState. Now we can call the grain states SetAll() method and we are set. Let's see the code:

public async Task ReadStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    try
    {
        var results = await _orchestrate.GetAsync(collectionName, key);

        var dict =     ((IGrainState)JsonConvert.DeserializeObject(results.Value.ToString(),       grainState.GetType())).AsDictionary();
        grainState.SetAll(dict);
    }
    catch (Exception ex)
    {
        Console.WriteLine("==> No record found in {0} collection for id {1}\n\r", new object[] { collectionName, key });
        WriteStateAsync(grainType, grainReference, grainState);
    }
}

Now if we have an exception thrown, it means that item does not exist in our collection. Because all grains always exist in Orleans, we will go ahead and write the item to our collection.

Speaking of writing, lets go ahead and look at the WriteStateAsync code:

public async Task WriteStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    try
    {
        var results = await _orchestrate.PutAsync(collectionName, key, grainState);
    }
    catch (Exception ex)
    {
        Console.WriteLine("==> Write failed in {0} collection for id {1}", new object[] { collectionName, key });
    }
}

And the ClearStateAsync:

public async Task ClearStateAsync(string grainType, GrainReference grainReference, GrainState grainState)
{
    var collectionName = grainState.GetType().Name;
    var key = grainReference.ToKeyString();

    await _orchestrate.DeleteAsync(collectionName, key, false);
}

The last method is the Close, with the Orchestrate.NET provider we can simply set our instance to null and be done. I will leave that code to you.

Wrap Up

And that is all there is to it. Check out the source code and sample project on github. I believe Orleans will have a place in your tool box and Orchestrate makes for a powerful persistence mechanism to pair with it.

If you are going to download and run the code yourself, make sure you create an app in Orchestrate, create ManagerState and EmployeeState collections, then grab your API key and put it at the appropriate place in the DevTestServerConfiguration.xml file.


Thursday, August 14, 2014

Project "Orleans" and Orchestrate.NET

Cloud applications are by default distributed and require parallel processing. Modern application users demand near real-time interaction and responses. Project "Orleans" is Microsoft's new framework for creating cloud (Azure) based distributed applications that meet these requirements. Based on established .Net code and practices, it brings the actor model to your toolbox.

Orleans

Orleans is a new middle tier framework that allows you to "cache" your business objects and their data. It accomplishes this with "grains". A grain is a single-threaded, encapsulated, light weight object that can communicate with other grains via asynchronous message passing. Grains are hosted in "Silos", typically one silo per server. Each silo in an application knows of the other silos and can pass grains and messages between them. Two main goals of Orleans is developer productivity and scaleability by default.

Developer Productivity

Productivity is achieved by providing a familiar environment for development. Grains are .Net objects with declared interfaces, this allows the grains to appear as simple remote objects that can be interacted with directly. Grains are also guaranteed to be single threaded, therefore the programmer never has to deal with locks or other synchronization methods to control access to shared resources. Grains are activated as needed, and if not in use can be garbage collected transparently. This makes grains behave like they are in a cache, "paged-in"/"paged-out" as required. The location of grains is transparent to the developer as well, programmers never need be concerned about which silo a grain is in, as all messaging is handled by the framework.

Scaleability

Fine-grain grains? Orleans makes it easy to break middle tier objects into small units. It can handle large numbers of actors with ease, millions or more, this allows Orleans to control what grains are active and where. Heavy loads on a specific section of grains will be load balanced automatically by the Orleans framework. Grains have logical end-points, with messaging multiplexed across a set of all-to-all physical connections via TCP, this allows a large number of addressable grains with low OS overhead. The Orleans run-time can schedule a large number of grains across a custom thread pool, allowing the framework to run at a high CPU utilization rate with high stability. The nature of messaging between actors allows programmers to develop non-blocking asynchronous code, allowing for a higher degree of parallelism and throughput without using multi-threading in the grains themselves.

Orchestrate.NET

Where does persistence come in? Orleans allows programmers to persist grain state via an integrated store. The grains will synchronize updates and guarantee that callers receive results only after the state has been successfully saved. This system is easily customized/extended and we will do so, and use Orchestrate.NET as the storage provider.

But not until the next post... Meantime you can read up on Orleans, Orleans @ build, Orchestrate and Orchestrate.NET.

Tuesday, May 6, 2014

Introducing Orchestrate.NET

I have had the pleasure of working on a new open source project called Orchestrate.NET, This project creates a wrapper for Orchestrate, the powerful new Database as a Service that allows you to combine multiple databases into a single service and query it with a simple API.

Once registered and signed in to the Orchestrate.io site, you can create Applications through the console. Each application comes with its own API key. Using this key and the Orchestrate.NET library it is easy to create collections, add/update/delete items, search using Lucene query strings as well as add events and object graphs.

With Orchestrate you pay by the number of data transactions, not by the size of your data set. With the first million transactions per month being free, its a great option for start ups without much traffic yet, and the enterprise with very large data sets. Orchestrate takes care of security, back ups and monitoring for you, leaving you free to concentrate on coding.

To add a reference to Orchestrate.NET to your project use the package manager:
PM> Install-Package Orchestrate.NET
or go to the repository on GitHub and get the code for yourself.

Once installed to create a collection use the following code (you have to insert an item to create a collection):
var orchestration = new Orchestrate("YOUR API KEY GOES HERE");
var result = orchestration.CreateCollection(<collectionname>, <itemid>, <item>);

To insert a new item (or update an existing one) you would do something like this:
var result = orchestration.Put(<collectionname>, <itemid>, <item>);

To retrieve an item by its Id:
var result = _orchestrate.Get(<collectionname>, <itemid>);

Please check out the documentation on GitHub and at Orchestrate.io for more in depth look at the capabilities available to you. Also please provide feedback, ideas for new features and pull requests as you can. We want to make this the best library possible.

I will be posting some more in depth looks at Orchestrate and how to use Orchestrate.NET to get the most out of it in your applications.