Tuesday, September 20, 2011

When the Reactive Framework meets F# 3.0

Few days ago I have been writing about asynchronous sequences in C#. This time we will see how easy it is to implement the same web crawler using Reactive Framework and a new F# 3.0 feature called Query Expressions. The original implementation of web crawler created by Tomas Petricek can be found here and my implementation is surprisingly similar:

let rec randomCrawl url = 
let visited = new System.Collections.Generic.HashSet<_>()

let rec loop url = obs {
if visited.Add(url) then
let! doc = (downloadDocument url) |> fromAsync
match doc with
| Some doc ->
yield url, getTitle doc
for link in extractLinks doc do
yield! loop link
| _ -> () }
loop url

rxquery {
for (url, title) in randomCrawl "http://news.bing.com" do
where (url.Contains("bing.com") |> not)
select title
take 10 into gr
iter (printfn "%s" gr)
}
|> ObservableExtensions.Subscribe |> ignore

There are two interesting things inside the code above. The first one is “obs { … } ” code construction. This is custom implementation of Computation Expression builder. obs is just a normal variable of a type that contains a special set of methods like: Bind, Delay, For, Combine, … and so on. The F# compiler translates the code inside the curly brackets into code calling those methods. In my case the whole expression returns the implementation of IObservable<T> type. This allows us to write imperative code representing observable source where each “yield” call returns next value (OnNext of subscribed observer is being called) and the “let!” keyword causes the program to wait for the values from the other observable source (let! keyword works like await keyword in C#). See the implementation of the builder class below:

type ObsBuiler() = 
member this.Zero () = Observable.Empty(Scheduler.CurrentThread)
member this.Yield v = Observable.Return(v, Scheduler.CurrentThread)
member this.Delay (f: _ -> IObservable<_>) = Observable.Defer(fun _ -> f())
member this.Combine (o1,o2) = Observable.Concat(o1,o2)
member this.For (s:seq<_>, body : _ -> IObservable<_>) = Observable.Concat(s.Select(body))
member this.YieldFrom a : IObservable<_> = a
member this.Bind ((o : IObservable<_>),(f : _ -> IObservable<_>)) = o.SelectMany(f)
member this.TryFinally((o : IObservable<_>),f : unit -> unit ) = Observable.Finally(o,fun _ -> f())
member this.TryWith((o : IObservable<_>),f : Exception -> IObservable<_> ) = Observable.Catch(o,fun ex -> f(ex))
member this.While (p,body) = Observable.While((fun () -> p()), body)
member this.Using (dis,body) = Observable.Using( (fun () -> dis), fun d -> body(d))

let obs = new ObsBuiler()

The second interesting part is “rxquery { … }” code construction. This time we are using a feature called Query Expressions introduced in F# 3.0 which was released few day ago during Build conference together with Visual Studio 11 and Windows 8. We can write queries very similar to LINQ queries and the F# compiler translate them into code calling methods like: where, select, groupBy, take and on so on. So it works like LINQ queries in C# but here we can extend the set of available methods arbitrarily !!! Look at Zip and ForkJoin methods below which are not available by default with QueryBuilder implementation working with IEnumerable<T> type. Let’s see the implementation of query builder class:

type RxQueryBuiler() =  
member this.For (s:IObservable<_>, body : _ -> IObservable<_>) = s.SelectMany(body)
[<CustomOperation("select", AllowIntoPattern=true)>]
member this.Select (s:IObservable<_>, [<ProjectionParameter>] selector : _ -> _) = s.Select(selector)
[<CustomOperation("where", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.Where (s:IObservable<_>, [<ProjectionParameter>] predicate : _ -> bool ) = s.Where(predicate)
[<CustomOperation("takeWhile", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.TakeWhile (s:IObservable<_>, [<ProjectionParameter>] predicate : _ -> bool ) = s.TakeWhile(predicate)
[<CustomOperation("take", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.Take (s:IObservable<_>, count) = s.Take(count)
[<CustomOperation("skipWhile", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.SkipWhile (s:IObservable<_>, [<ProjectionParameter>] predicate : _ -> bool ) = s.SkipWhile(predicate)
[<CustomOperation("skip", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.Skip (s:IObservable<_>, count) = s.Skip(count)
member this.Zero () = Observable.Empty(Scheduler.CurrentThread)
member this.Yield (value) = Observable.Return(value)
[<CustomOperation("count")>]
member this.Count (s:IObservable<_>) = Observable.Count(s)
[<CustomOperation("all")>]
member this.All (s:IObservable<_>, [<ProjectionParameter>] predicate : _ -> bool ) = s.All(new Func<_,bool>(predicate))
[<CustomOperation("contains")>]
member this.Contains (s:IObservable<_>, key) = s.Contains(key)
[<CustomOperation("distinct", MaintainsVariableSpace=true, AllowIntoPattern=true)>]
member this.Distinct (s:IObservable<_>) = s.Distinct()
[<CustomOperation("exactlyOne")>]
member this.ExactlyOne (s:IObservable<_>) = s.Single()
[<CustomOperation("exactlyOneOrDefault")>]
member this.ExactlyOneOrDefault (s:IObservable<_>) = s.SingleOrDefault()
[<CustomOperation("find")>]
member this.Find (s:IObservable<_>, [<ProjectionParameter>] predicate : _ -> bool) = s.First(new Func<_,bool>(predicate))
[<CustomOperation("head")>]
member this.Head (s:IObservable<_>) = s.First()
[<CustomOperation("headOrDefault")>]
member this.HeadOrDefault (s:IObservable<_>) = s.FirstOrDefault()
[<CustomOperation("last")>]
member this.Last (s:IObservable<_>) = s.Last()
[<CustomOperation("lastOrDefault")>]
member this.LastOrDefault (s:IObservable<_>) = s.LastOrDefault()
[<CustomOperation("maxBy")>]
member this.MaxBy (s:IObservable<'a>, [<ProjectionParameter>] valueSelector : 'a -> 'b) = s.MaxBy(new Func<'a,'b>(valueSelector))
[<CustomOperation("minBy")>]
member this.MinBy (s:IObservable<'
a>, [<ProjectionParameter>] valueSelector : 'a -> 'b) = s.MinBy(new Func<'a,'b>(valueSelector))
[<CustomOperation("nth")>]
member this.Nth (s:IObservable<'a>, index ) = s.ElementAt(index)
[<CustomOperation("sumBy")>]
member inline this.SumBy (s:IObservable<_>,[<ProjectionParameter>] valueSelector : _ -> _) = s.Select(valueSelector).Aggregate(Unchecked.defaultof<_>, new Func<_,_,_>( fun a b -> a + b))
[<CustomOperation("groupBy", AllowIntoPattern=true)>]
member this.GroupBy (s:IObservable<_>,[<ProjectionParameter>] keySelector : _ -> _) = s.GroupBy(new Func<_,_>(keySelector))
[<CustomOperation("groupValBy", AllowIntoPattern=true)>]
member this.GroupValBy (s:IObservable<_>,[<ProjectionParameter>] resultSelector : _ -> _,[<ProjectionParameter>] keySelector : _ -> _) = s.GroupBy(new Func<_,_>(keySelector),new Func<_,_>(resultSelector))
[<CustomOperation("join", IsLikeJoin=true)>]
member this.Join (s1:IObservable<_>,s2:IObservable<_>, [<ProjectionParameter>] s1KeySelector : _ -> _,[<ProjectionParameter>] s2KeySelector : _ -> _,[<ProjectionParameter>] resultSelector : _ -> _) = s1.Join(s2,new Func<_,_>(s1KeySelector),new Func<_,_>(s2KeySelector),new Func<_,_,_>(resultSelector))
[<CustomOperation("groupJoin", AllowIntoPattern=true)>]
member this.GroupJoin (s1:IObservable<_>,s2:IObservable<_>, [<ProjectionParameter>] s1KeySelector : _ -> _,[<ProjectionParameter>] s2KeySelector : _ -> _,[<ProjectionParameter>] resultSelector : _ -> _) = s1.GroupJoin(s2,new Func<_,_>(s1KeySelector),new Func<_,_>(s2KeySelector),new Func<_,_,_>(resultSelector))
[<CustomOperation("zip", IsLikeZip=true)>]
member this.Zip (s1:IObservable<_>,s2:IObservable<_>,[<ProjectionParameter>] resultSelector : _ -> _) = s1.Zip(s2,new Func<_,_,_>(resultSelector))
[<CustomOperation("forkJoin", IsLikeZip=true)>]
member this.ForkJoin (s1:IObservable<_>,s2:IObservable<_>,[<ProjectionParameter>] resultSelector : _ -> _) = s1.ForkJoin(s2,new Func<_,_,_>(resultSelector))
[<CustomOperation("iter")>]
member this.Iter(s:IObservable<_>, [<ProjectionParameter>] selector : _ -> _) = s.Do(selector)

let rxquery = new RxQueryBuiler()
 

Thursday, September 15, 2011

Design-time support for Caliburn.Micro

Recently me together with Bartek Pampuch have been wondering if it’s possible to fire Caliburn.Micro’s binding process at design-time. Caliburn.Micro is a really great framework (just take a look at some of our multitouch apps based on this framework and BFSharp). You just name the controls appropriately and all magic happens automatically. Controls are bound to the View Model’s properties and methods for you. The problem is that it is all happing at run time not at design time. When you are preparing the form in the Visual Studio or Expression Blend you aren’t able to see how the form will look like with bound data. In some cases such feature would very useful, especially when we want to use the sample data generated by Expression Blend or Visual Studio designers. In this post we will show you how to run Caliburn.Micro’s conventions based binding mechanism at design time.

Firstly, we discovered that we can change the objects tree representing the screen and that change is not persisted back to xaml file. You can read more about this feature in my previous post. Secondly, we tried to run Caliburn.Micro’s binding process to see what happens. What was really amazing is that after first try all just started working smoothly! The framework itself is implemented so well :)

One of the samples provided with Caliburn.Micro release is a very simple application called GameLibrary. AddGameView.xaml file defines the screen that looks like this:

image

After setting up sample data generated by Visual Studio and setting attached property 'DesignTime.Enable=”True” the same form looks like this:

clip_image001

In the simplest scenario all we need to run Caliburn.Micro binding process at design time is setting design time data context and enabling binding via Enable attached property. In sample above we are setting sample data generated by Visual Studio because the AddGameViewModel class doesn’t provide default constructor. Of course we could always add a default constructor in code and define view model instance in xaml file.

I chose this particular form intentionally because it contains the control that hasn’t defined custom binding convention by default. The Rating control is responsible for displaying Rating property value using stars. Rating property value is 0.8 so 4 stars should be displayed. In such cases where we are using controls with custom binding convention we need to register those conventions at design time. Any application using Caliburn.Micro has a type inherited from Bootstrapper type. This type is a starting point of our application and the instance of that type very often is defined as a resource inside App.xaml file. In fact Caliburn.Micro framework is already prepared for design time scenarios because Bootstrapper type contains virtual method called StartDesignTime. Let’s override this method and register appropriate convention:


public class Bootstrapper
{
public Bootstrapper()
{
if (Execute.InDesignMode) StartDesignTime(); else StartRuntime();
}
}

public class Bootstrapper<TRootModel> : Bootstrapper
{
}

public class AppBootstrapper : Bootstrapper<IShell>
{
private static bool isInitializedInDesignTime = false;

protected override void StartDesignTime()
{
base.StartDesignTime();

if (isInitializedInDesignTime)
return;
isInitializedInDesignTime = true;

ConfigureConventions();
}


void ConfigureConventions()
{
ConventionManager.AddElementConvention<Rating>(Rating.ValueProperty, "Value", "ValueChanged");
}
}

The instance of AppBootstapper type can be created many times at design time so boolean flag ensures that the conventions will be registered only once. When we compile the project and reopen the form we will see something like this:

image

This form represents a very simple scenario where the view model type has only properties of simple types like: string, double. boolean. It doesn’t contain any property of collection type or other view model type. In such scenarios Caliburn.Micro can find the appropriate type of view (control type) based on the type of view model. Let’s see another form called ResultsView.xaml to demonstrate this case:

image

This is a typical problem when we work with Caliburn.Micro and we cannot see anything at design time because the control is entirely collapsed :). Let’s see what happen after connecting sample data.

image

The list contains some elements but the font color is white so we aren’t able to read it because of the default Visual Studio background color. We can try to open this form inside Expression Blend where the background is dark or we can use custom design time attributes presented in the previous post.

image

Now we can see that Caliburn.Micro is not able to find view for view model type representing list item. It’s because we are using sample data mechanism from Visual Studio which generates dynamic type _.di0.GameLibrary.ViewModels.IndividualResultViewModel with the shape of the original view model type GameLibrary.ViewModels.IndividualResultViewModel. We need to change the way Caliburn.Micro is searching for view type based on specified view model type.

protected override void StartDesignTime()
{
base.StartDesignTime();

if (isInitializedInDesignTime)
return;
isInitializedInDesignTime = true;

ConfigureConventions();

AssemblySource.Instance.AddRange(new[] { typeof(App).Assembly });

var originalLocateTypeForModelType = ViewLocator.LocateTypeForModelType;
Func<Type, bool> isDesignTimeType = type => type.Assembly.IsDynamic;
ViewLocator.LocateTypeForModelType = (modelType, displayLocation, context) =>
{
var type = originalLocateTypeForModelType(modelType, displayLocation, context);
if (type == null && isDesignTimeType(modelType))
{
if (modelType.Name == "IndividualResultViewModel")
{
type = typeof(IndividualResultView);
}
}
return type;
};

IoC.GetInstance = base.GetInstance;
IoC.GetAllInstances = base.GetAllInstances;
}

Finally the list of items and generated sample data look like this:

image

Attached property is not the most convenient way of extending the designer functionality because we need to write xaml code. I tried to rewrite attached property to custom behavior so we could use dra&drop instead of writing the code. The problem is that behaviors code is not executed at design time. But I have good news. If you are creating UI with Expression Blend you can use our attached property on the property grid like a normal property.

image

It was possible thanks to the usage of attribute called AttachedPropertyBrowsableForTypeAttribute decorating attached property. Everything that was presented so far works both in Silverlight and WPF environments. It’s time to reveal how the magic works.



public static class DesignTime
{
public static DependencyProperty EnableProperty =
DependencyProperty.RegisterAttached(
"Enable",
typeof(bool),
typeof(DesignTime),
new PropertyMetadata(new PropertyChangedCallback(EnableChanged)));

#if !SILVERLIGHT && !WP7
[AttachedPropertyBrowsableForTypeAttribute(typeof(DependencyObject))]
#endif
public static bool GetEnable(DependencyObject dependencyObject)
{
return (bool)dependencyObject.GetValue(EnableProperty);
}


public static void SetEnable(DependencyObject dependencyObject, bool value)
{
dependencyObject.SetValue(EnableProperty, value);
}

static void EnableChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
{
if (!Execute.InDesignMode)
return;

BindingOperations.SetBinding(d, DataContextProperty, (bool)e.NewValue ? new Binding() : null);
}

private static readonly DependencyProperty DataContextProperty =
DependencyProperty.RegisterAttached(
"DataContext",
typeof(object),
typeof(DesignTime),
new PropertyMetadata(new PropertyChangedCallback(DataContextChanged))
);

private static void DataContextChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
{
if (!Execute.InDesignMode)
return;

object enable = d.GetValue(EnableProperty);
if (enable == null || ((bool)enable) == false || e.NewValue == null)
return;

var fe = d as FrameworkElement;
if (fe == null)
return;

ViewModelBinder.Bind(e.NewValue, d, string.IsNullOrEmpty(fe.Name) ? fe.GetHashCode().ToString() : fe.Name);
}
}

I hope you find this solution useful.

Download

Custom design-time attributes in Silverlight and WPF designer

We all know the standard design-time attributes in the Silverlight designer like d:DataContext, d:DesignWidth or d:DesignHeight. Let’s see how easy we can add some new attributes.

image
using System.ComponentModel;
using System.Windows;

namespace DesignTimeProperties
{
public static class d
{
static bool? inDesignMode;

/// <summary>
/// Indicates whether or not the framework is in design-time mode. (Caliburn.Micro implementation)
/// </summary>
private static bool InDesignMode
{
get
{
if (inDesignMode == null)
{
var prop = DesignerProperties.IsInDesignModeProperty;
inDesignMode = (bool)DependencyPropertyDescriptor.FromProperty(prop, typeof(FrameworkElement)).Metadata.DefaultValue;

if (!inDesignMode.GetValueOrDefault(false) && System.Diagnostics.Process.GetCurrentProcess()
.ProcessName.StartsWith("devenv", System.StringComparison.Ordinal))
inDesignMode = true;
}

return inDesignMode.GetValueOrDefault(false);
}
}

public static DependencyProperty BackgroundProperty = DependencyProperty.RegisterAttached(
"Background", typeof(System.Windows.Media.Brush), typeof(d),
new PropertyMetadata(new PropertyChangedCallback(BackgroundChanged)));

public static System.Windows.Media.Brush GetBackground(DependencyObject dependencyObject)
{
return (System.Windows.Media.Brush)dependencyObject.GetValue(BackgroundProperty);
}
public static void SetBackground(DependencyObject dependencyObject, System.Windows.Media.Brush value)
{
dependencyObject.SetValue(BackgroundProperty, value);
}
private static void BackgroundChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
{
if (!InDesignMode)
return;

d.GetType().GetProperty("Background").SetValue(d, e.NewValue, null);
}
}
}

Great, but what about many others very useful properties ? Do we need to write them all manually? No, we don’t. I have prepared T4 template that generates code for all properties of all controls available in WPF and Silverlight assemblies. When you look carefully at the code above you will find that I am using the reflection to set the value of the property. It’s because some properties like Background are defined many times in many different controls so instead of checking the actual control type I assume that the control contains appropriate property. This assumption simplifies the implementation very much. And of course the property can be set only in design time when we are editing the form inside Expression Blend or Visual Studio. T4 template code analyzes all properties from all controls and properties with the same names but different types are skipped during generation process.

Download

Sunday, September 4, 2011

Programming with C# asynchronous sequences

Tomas Petricek in his last blog post titled “Programming with F# asynchronous sequences” presents F# implementation of something called asynchronous sequences. In this post I will show you how the same concept can be implemented in C#. Let’s look at the sample code below to better understand what the asynchronous sequence is:

IEnumerable<...> AsyncSeq()
{
yield return "Hello";
await TaskEx.Delay(100);
yield return "world!";
}

Asynchronous sequences is a code that produces the sequence of values generated on demand (this is how the IEnumerable interface can be interpreted) but additionally does some asynchronous work during the evaluation process (await keyword). Every time the client of asynchronous sequence calls MoveNext method, next value is being evaluated. The key feature here is that the client decides when to produce next value and when to stop the processing.

There are two problems with such an implementation of asynchronous sequence. Sequences in .Net world are represented with IEnumerable interface, but the interface allows only synchronous processing. Since the MoveNext method returns bool value in the interface implementation, we need to immediately decide whether the next value can be produced or not. In the asynchronous processing it can take a few minutes or even hours to provide such information. The second problem is that so far we cannot mix together await keyword (Async Ctp) with yield return/yield break keywords inside the same method body. My solution resolves those two problems and the above sequence can be implemented the fallowing way:

IEnumerable<AsyncSeqItem<string>> AsyncSeq()
{
yield return "Hello";
yield return TaskEx.Delay(100);
yield return "world!";
}

public enum AsyncSeqItemMode
{
Value, Task, Sequence
}

public class AsyncSeqItem<T>
{
public AsyncSeqItemMode Mode { get; private set; }
public T Value { get; private set; }
public Task Task { get; private set; }
public IEnumerable<AsyncSeqItem<T>> Seq { get; private set; }

public AsyncSeqItem(T value)
{
Value = value;
Mode = AsyncSeqItemMode.Value;
}

public AsyncSeqItem(Task task)
{
Task = task;
Mode = AsyncSeqItemMode.Task;
}

public AsyncSeqItem(IEnumerable<AsyncSeqItem<T>> seq)
{
Seq = seq;
Mode = AsyncSeqItemMode.Sequence;
}

public static implicit operator AsyncSeqItem<T>(T value)
{
return new AsyncSeqItem<T>(value);
}

public static implicit operator AsyncSeqItem<T>(Task task)
{
return new AsyncSeqItem<T>(task);
}
}

AsyncSeqItem represents one of three following values:

  • Value – next value generated by the sequence
  • Task – some asynchronous work that needs to be awaited for before going forward
  • Sequence – it’s used with recursive calls and it means that we want to use tail recursion

There are two ways of consuming such sequence in the client:

public static class AsyncSeqExtensions
{
public static IEnumerable<Task<Option<T>>> ToTaskEnumerable<T>(this IEnumerable<AsyncSeqItem<T>> seq, bool continueOnCapturedContext = true)
{ ... }

public static IAsyncEnumerable<T> ToAsyncEnumerable<T>(this IEnumerable<AsyncSeqItem<T>> seq, bool continueOnCapturedContext = true)
{ ... }
}

public class Option<T>
{
public T Value { get; private set; }
public bool HasValue { get; private set; }

public Option()
{
HasValue = false;
}

public Option(T value)
{
Value = value;
HasValue = true;
}

public static implicit operator Option<T>(T value)
{
return new Option<T>(value);
}
}

In the first approach we are calling ToAsyncEnumerable extension method returning the sequence of tasks. Each task wraps special type called Option<T> which can be used similarly to Nullable<T> type except that it works with value and reference types. Returning task with option object without the value means that we reached the end of the sequence. I also provide few standard LINQ operators built on the top of such a sequence semantic:

public static class AsyncSeqExtensions
{
async private static Task ForEachTaskImpl<T>(this IEnumerable<Task<Option<T>>> seq, Action<Task<Option<T>>> action)
{
foreach (var task in seq)
{
await task;
action(task);
}
}
public static Task ForEachTask<T>(this IEnumerable<Task<Option<T>>> seq, Action<Task<Option<T>>> action)
{
return ForEachTaskImpl(seq, action);
}

public static Task ForEach<T>(this IEnumerable<Task<Option<T>>> seq, Action<T> action)
{
return seq.ForEachTask(task =>
{
if(task.Result.HasValue)
action(task.Result.Value);
});
}

async private static Task<T[]> ToArrayImpl<T>(IEnumerable<Task<Option<T>>> seq)
{
var list = new List<T>();
await seq.ForEach(v => list.Add(v));
return list.ToArray();
}
public static Task<T[]> ToArray<T>(this IEnumerable<Task<Option<T>>> seq)
{
return ToArrayImpl(seq);
}

public static IEnumerable<Task<Option<TResult>>> Select<T, TResult>(this IEnumerable<Task<Option<T>>> source,
Func<T,TResult> selector) { ... }

public static IEnumerable<Task<Option<T>>> Where<T>(this IEnumerable<Task<Option<T>>> source,
Func<T, bool> predicate) { ... }

public static IEnumerable<Task<Option<T>>> Take<T>(this IEnumerable<Task<Option<T>>> source,
int count) { ... }

...
}

Returning additional task object at the end of a sequence with special value allows us to use standard IEnumerable<T> interface but it’s a little bit inconvenient. In the second approach we use the IAsyncEnumerable interface from the Reactive Framework library released some time ago.

public interface IAsyncEnumerable<out T>
{
IAsyncEnumerator<T> GetEnumerator();
}

public interface IAsyncEnumerator<out T> : IDisposable
{
Task<bool> MoveNext();
T Current { get; }
}

public static class AsyncEnumerable
{
public static IAsyncEnumerable<TResult> Select<TSource, TResult>(IAsyncEnumerable<TSource> source,
Func<TSource, TResult> selector) { ... }

public static IAsyncEnumerable<TSource> Where<TSource>(IAsyncEnumerable<TSource> source,
Func<TSource, bool> predicate) { ... }

public static IAsyncEnumerable<TSource> Take<TSource>(IAsyncEnumerable<TSource> source,
int n) { ... }
}

This interface perfectly represents the semantic of asynchronous sequence. Rx library also provides many standard LINQ operations like: Where, Select, Take, Sum, First and so on. This allows us to write almost any LINQ query executing on the top of asynchronous sequence.

Now let’s summarize what we achieved so far. We can write imperative code implementing asynchronous sequence. We can use extension method to create one of two asynchronous sequence representations. Finally we can iterate through all items in such a sequence or we can build a new sequence object using LINQ operators.

The C# version of the web crawler presented in Tomas Petricek’s blog post could look like this:

public static class AsyncSeqSample
{
async public static Task CrawlBingUsingAsyncEnumerable()
{
await RandomCrawl("http://news.bing.com")
.ToAsyncEnumerable()
.Where(t => !t.Item1.Contains("bing.com"))
.Select(t => t.Item2)
.Take(10)
.ForEach(Console.WriteLine);

Console.WriteLine("the end...");
}

async public static Task CrawlBingUsingTaskEnumerable()
{
await RandomCrawl("http://news.bing.com")
.ToTaskEnumerable()
.Where(t => !t.Item1.Contains("bing.com"))
.Select(t => t.Item2)
.Take(10)
.ForEach(Console.WriteLine);

Console.WriteLine("the end...");
}

public static IEnumerable<AsyncSeqItem<Tuple<string, string>>> RandomCrawl(string url)
{
return RandomCrawlLoop(url, new HashSet<string>());
}

private static IEnumerable<AsyncSeqItem<Tuple<string,string>>> RandomCrawlLoop(string url, HashSet<string> visited)
{
if (visited.Add(url))
{
var downloadTask = DownloadDocument(url);
yield return downloadTask;
if (downloadTask.Result.HasValue)
{
var doc = downloadTask.Result.Value;
yield return Tuple.Create(url, GetTitle(doc));
foreach (var link in ExtractLinks(doc))
{
foreach (var l in RandomCrawlLoop(link, visited))
{
yield return l;
}
}
}
}
}

private static string[] ExtractLinks(HtmlDocument doc)
{
try
{
var q = from a in doc.DocumentNode.SelectNodes("//a")
where a.Attributes.Contains("href")
let href = a.Attributes["href"].Value
where href.StartsWith("http://")
let endl = href.IndexOf('?')
select endl > 0 ? href.Substring(0, endl) : href;

return q.ToArray();
}
catch
{
return new string[0];
}
}

async private static Task<Option<HtmlDocument>> DownloadDocument(string url)
{
try
{
var client = new WebClient();
var html = await client.DownloadStringTaskAsync(url);
var doc = new HtmlDocument();
doc.LoadHtml(html);
return new Option<HtmlDocument>(doc);
}
catch (Exception)
{
return new Option<HtmlDocument>();
}
}

private static string GetTitle(HtmlDocument doc)
{
var title = doc.DocumentNode.SelectSingleNode("//title");
return title != null ? title.InnerText.Trim() : "Untitled";
}
}

Now let’s see how ToAsyncEnumerable and ToTaskEnumerable methods have been implemented:

public static class AsyncSeqExtensions
{
public static IAsyncEnumerable<T> ToAsyncEnumerable<T>(this IEnumerable<AsyncSeqItem<T>> seq, bool continueOnCapturedContext = true)
{
if (seq == null) throw new ArgumentNullException("seq");

return new AnonymousAsyncEnumerable<T>(() =>
{
var enumerator = seq.ToTaskEnumerable(continueOnCapturedContext).GetEnumerator();
seq = null; // holding reference to seq parameter introduces memory leaks when asynchronous sequence uses recursive calls
TaskCompletionSource<bool> currentTcs = null;
Task<Option<T>> currentTask = null;

return new AnonymousAsyncEnumerator<T>(
() =>
{
currentTcs = new TaskCompletionSource<bool>();

if (CheckEndOfSeq(currentTask) == false)
{
currentTcs.SetResult(false);
return currentTcs.Task;
}

enumerator.MoveNext();

enumerator.Current.ContinueWith(t =>
{
if (t.IsFaulted)
{
currentTcs.SetException(t.Exception);
}
else
{
if (!t.Result.HasValue)
{
currentTcs.SetResult(false);
}
else
{
currentTask = t;
currentTcs.SetResult(true);
}
}
});

return currentTcs.Task;
},
() => currentTask.Result.Value
);
});
}

public static IEnumerable<Task<Option<T>>> ToTaskEnumerable<T>(this IEnumerable<AsyncSeqItem<T>> seq, bool continueOnCapturedContext = true)
{
if (seq == null) throw new ArgumentNullException("seq");

return new AnonymousEnumerable<Task<Option<T>>>(() =>
{
var synchronizationContext = continueOnCapturedContext ? SynchronizationContext.Current : null;

var enumerator = seq.GetEnumerator();
seq = null; // holding reference to seq parameter introduces memory leaks when asynchronous sequence uses recursive calls

TaskCompletionSource<Option<T>> currentTcs = null;

return new AnonymousEnumerator<Task<Option<T>>>(
() =>
{
if (CheckEndOfSeq(currentTcs) == false)
return false;

currentTcs = new TaskCompletionSource<Option<T>>();

Action moveNext = null;
moveNext = () =>
{
Start:

bool b;

try
{
b = enumerator.MoveNext();
}
catch (Exception exception)
{
currentTcs.SetException(exception);
return;
}

if (b == false)
{
currentTcs.SetResult(new Option<T>());
}
else
{
var c = enumerator.Current;
if (c.Mode == AsyncSeqItemMode.Value)
{
currentTcs.SetResult(c.Value);
}
else if (c.Mode == AsyncSeqItemMode.Task)
{
if (synchronizationContext != null)
c.Task.ContinueWith(_ => synchronizationContext.Post(s => ((Action)s)(), moveNext));
else
c.Task.ContinueWith(_ => moveNext());
}
else if (c.Mode == AsyncSeqItemMode.Sequence)
{
enumerator = c.Seq.GetEnumerator();
goto Start;
}
}
};

moveNext();

return true;
},
() => currentTcs.Task
);
});
}
}

As you can see the implementation is really simple but the whole concept of asynchronous sequence is very powerful.

Download