Sync Framework 4.0

May 20, 2011 at 9:55 PM
Edited May 20, 2011 at 9:56 PM

As far as I see is no big difference from  Sync Framework 4.0 : Anchor, tombstoning, service oriented Sync, what's the point/direction? Why not Sync Framework since use (almost)standard protocol ODataSync. And also client platform can be independent? Confusing totally your strategy...

Coordinator
May 22, 2011 at 2:11 AM

Hi Cristoph

Thanks for bringing this up. In a 1 hour  talk, it’s hard to bring up all the points and caveats. Being Tech Ed, I wanted to stay on the actionable things people can do today. For instance, I really wanted to spend more time on the local data storage options. Three’s not perfect solution today, and wanted to talk more about the various options and debates (Relational or Object) for instance.

Sync Concepts

I’ve had the pleasure of working on many of our sync products, including Sync Services for ADO.net, Sync Framework and other components like SQL Server Compact (SQLce) and some early work on sync enabling Astoria offline which is why some of the concepts discussed, like Anchors and Tombstones look familiar. I wanted to make sure we are leveraging the great work done by various teams. That said, since the talk was at Tech Ed, as opposed to Mix or PDC, I felt I needed to focus on what developers can build today, but it’s actually not that simple, so let me explain a bit.

Sync Framework:

As the Sync Framework team discussed here they’re focused on cloud sync features and unfortunately have indefinitely suspended the release of future versions of the Sync Framework.

OData Caching

There is some great work happening here, which is leveraging the designs of the Sync Framework, and I do encourage people to take a look and give it a try. In fact, OData can be incorporated into the pattern I proposed. At least for the download. A this point, OData doesn’t address changes made locally, however, I’m not sure it should. See below for “Sync Isn’t Service Friendly”. The main reason I didn’t directly use it for the talk was it’s not yet shipping and wanted to focus on what people could do today, but there’s more:

Messaging & Services

The main concept I wanted to present here was the hybrid approach to Active Caching, Passive Caching and “Messaging”*. There were two main factors that contributed to this evolved design pattern:

  • Sync isn’t service friendly
  • How to minimize latency of data when online

Sync isn’t service friendly

After many years of feedback from customers that “Sync” is neither service friendly, nor can they typically get those services exposed directly over their data, I’ve been wanting to take a different approach. If building a system from scratch, one could likely build a “sync service” exposing their data with anchors and/or knowledge. The challenge I continually heard was very rarely did people have green field projects, they can’t expose their data directly to services as they typically aggregate data from different systems or reshape the data, or they just had fundamental issues with the 1:1 mapping of their database and the data returned from the services. The bigger challenge was the data changed locally and how it gets sent up to the server. In Sync Services for ADO.net, we had several client and server events for dealing with conflicts. The problem is this didn’t mesh with the rest of their existing services for uploading changes, which also dealt with conflicts.

Trying to convince the DBA, systems/SOA architect, and various other stake holders in a company to make these sync changes to their service architecture was a constant challenge It seemed a more digestible approach to enhance existing services with anchors. If a service already returns customers, appointments, orders, products, states, etc. is it easier to simply add an “anchor” parameter to those existing services? Then, in the result, add a collection of deletes, if needed?

Using this approach, we still maintain the core concepts of sync, anchors, and even “knowledge”, but can incorporate it into existing services.

Latency of data when online

The traditional sync pattern we had proposed was to always work offline, then sync as often as you felt the data would change. Hmmm, how to predict changes? Reference data, not a big deal. There are various approaches to trigger a refresh of infrequently queried data. However, what to do with those Appointments, Reservations, or other things that may not change that often, but when they do, you want to get the updates to your users ASAP. Or, more specifically, when they want to view that data, if they’re online, to provide them the latest view.

You could just “sync” very often, which means when you’re connected you’re probably putting a bigger load on your system as you keep asking “what changed”, “what changed”, “what changed”. Even if you’re users aren’t actually looking at the data. When you’re online, users are perfectly fine with the round-trip response time. This would be difficult to do in a sync world without a duplex type connection, which kills scalability.

So, this is where we took a slightly different approach. We still do the periodic “sync” of data. Let’s say every 15 minutes. But, when you’re online, and your users ask for a specific Appointment, or even the list of Appointments, why not ask the server like you normally would? Then, just cache the results, on top of the previous baseline sync.

Now, there are still plenty of optimizations here, but this pattern of utilizing requests/”messages”, and routing them, caching them locally and surfacing them to the view model felt like a better approach that’s addressed much of the feedback we’ve received over the sync patterns. But, then, that’s the point, to solicit feedback and similar to MVVM, evolve the pattern.

Messages*

I will add that I do struggle with calling the requests, messages as they contain behavior. And perhaps that’s what they should be called, requests, with a request manager.

Continued Feedback

We appreciate the feedback, as we believe this is an important space.

Thanks
Steve

May 23, 2011 at 10:54 AM
Edited May 23, 2011 at 10:55 AM

Thank you Steve for very detailed answer, the approach is indeed different than SyncFramework way where user must work always with data from offline storage.

I'm completely agree with "Sync isn’t service friendly" and I'm looking further if this new approach can be expanded and can be used by various clients like SyncFRW 4.0 do: WP7, WPF,WinForms and non MS clients. Do you have any plans to make progress in this direction? I mean this demo, was just to show what you can do today for this scenarios offline/online or just a preview of a next solution for this scenarios?(I was not a TechEd so maybe you explained there the future about this?)

May 23, 2011 at 2:43 PM

Thanks for the clarification, Steve.   I was wondering about the current picture for building OCCA Silverlight applications given the recent re-focus of the Sync Framework Team to cloud based sync.  Based on your extensive experience in building sync solutions, what features/ideas do you feel are still "lacking"/immature and would require more attention in the current solution you've proposed?   You've mentioned the choice of local storage engine as an option that warrants further investigation, but are there any other further architectural issues that need to be addressed client/server side?   We are quite interested in trying out these techniques to build an off-line extension for a part of our Enterprise Asset Management System.

Thanks

Coordinator
May 24, 2011 at 4:21 PM

Pattern evolvement:
This was built as a demo specifically for Tech Ed. It was actaully a session I was asked to cover that was proposed by someone else and given the topic, felt I needed to give it the best position possible. So, unfortunately, it's not a product/offering we are actively developing. ...as much as I'd like to. That said, Jason Shaver, the PM on my team that did the majority of the development and kept pushing for the messaging concepts would like to keep it alive and hopefully as we build new product demos, we'll evolve and incorporate polish to the solution.

At some point, I beleive they'll be making the recording of Tech Ed available, so at least you can get the explinations to the slides and demos.

For lacking/immaturity, that's a good question. I like where the pattern was going as it bridged many of the challenges we've faced in the past. In reality, I think that's the hardes part. The detail implementation is a matter of rinse & repeat. The things I wish we had more time to flush out are:

  • Digging deeper into the object stores for local cahcing.
    We really needed more time to figure out how to manage the speed, indexing and how to store objects locally. We took the approach to store all objects in one collection and we had some concerns that we didn't get to flush out. Is it better to store in one collection, or individual collections
  • Overall framework evolvement
    We did try to build a robust pattern for the base classes. As with most of these efforts, the first pass isn't always the best and I think we learned a lot.  Finding the balance between simplicity and flexibility is a tough balance. Having the time to factor out the base classes, events and even some of the names were the places I think we can do better

So, overall, I think it's heading in the right direction and just needs some bake time and polish to figure out the quirks. Hopefully we'll have some time to do this for our next app building pass, but also very interested in what others think and are doing here.

What gaps do you see with this pattern?

Steve

May 24, 2011 at 4:39 PM

Thanks again for the reply.  At this time we are considering using something like your solution but we also need to investigate the possibility of building a HTML 5 off-line extension to our Silverlight Enterprise Asset Management System.  Some customers seem keen on an offline solution that supports iPad's.  That sort of puts a spanner in the works of a Silverlight solution :-(  Currently I'm trying to understand the offline support picture for HTML 5, but it all seems very confusing to me at this point in time.  There doesn't seem to widespread agreement on the adoption of a WEB SQL Database solution and I'm not sure the IndexDB API will be enough for an offline solution. It also implies a new technology stack for our development team to master.

So I'm not quite sure what route we are going to follow yet, but if we decides to stick with Silverlight we will for sure post back with any learnings we discovered in trying to get this production ready.

Coordinator
May 26, 2011 at 6:27 AM

About using HTML 5 offline,

It seems to me that the HTML 5 story for offline data is a bit too week right now.  If you want to work across most browsers, you don't have an option besides the 5mb (10 in IE) localstorage system.  The IndexDB API seems like the way everything will end up, but it will be a bit until we can rely on end users having it.  

Jason R. Shaver

May 26, 2011 at 6:49 AM

Jason, I agree.  It will be a while before the browser stacks have standardized across the IndexedDB API.  That unfortunately leaves us a with a bad picture for cross-platform .NET development where offline support is a pre-requisite.  The Mono stack seems the only way to go if you want to leverage your existing C# skillset, but it has quite a cost associated with it and the current uncertainty around the technology whilst Xamarin gets going.  So if you need to support a lot of different devices, you either need to go native or wait and hope that the Xamarin technology stack matures quickly.