Solutionizing .NET (Keith Dahlby)

posh-git: A PowerShell Environment for Git

2010-03-15T14:31:00Z

I've been meaning to put together a post about this for a few days, but Mark Embling beat me to it.

I'm a big fan of PowerShell and Git, but had never bothered using them together until Jeremy Skinner's post about adding tab completion pushed me over the edge. Mark's post has more details, but the short version is that there's a Google Group and GitHub repository that we will be using going forward in an effort to make Git and PowerShell work better together.

Over the coming weeks I'll have more to say about what we've been up to, but if you're at all interested in Git on PowerShell or have your own tips or tricks you'd like to share, please leave a comment or join the group.

SPWeb.AssociatedGroups.Contains Lies

2010-01-27T11:00:00Z

While working on SPExLib (several months ago), I revisited this post, which presented a functional approach to a solution Adam describes here. Both posts include logic to add an SPWeb group association, which most simply could look something like this:

SPGroup group = web.SiteGroups[groupName];
if (!web.AssociatedGroups.Contains(group))
{
    web.AssociatedGroups.Add(group);
    web.Update();
}

While testing on a few groups, I noticed that the Contains() call lies, always returning false. This behavior can also be verified with PowerShell:

PS > $w.AssociatedGroups | ?{ $_.Name -eq 'Designers' } | select Name

Name
----
Designers

PS > $g = $w.SiteGroups['Designers']
PS > $w.AssociatedGroups.Contains($g)
False

Of course, it's not actually lying—it just doesn't do what we expect. Behind the scenes, AssociatedGroups is implemented as a simple List<SPGroup> that is populated with group objects retrieved by IDs stored in the SPWeb's vti_associategroups property. The problem is that List<T>.Contains() uses EqualityComparer<T>.Default to find a suitable match, which defaults to reference equality for reference types like SPGroup that don't implement IEquatable<T> or override Equals().

To get around this, SPExLib provides a few extension methods to make group collections and SPWeb.AssociatedGroups easier to work with and more closely obey the Principle of Least Surprise:

public static bool NameEquals(this SPGroup group, string name)
{
    return string.Equals(group.Name, name, StringComparison.OrdinalIgnoreCase);
}

public static bool Contains(this SPGroupCollection groups, string name)
{
    return groups.Any<SPGroup>(group => group.NameEquals(name));
}

public static bool HasGroupAssociation(this SPWeb web, string name)
{
    return web.AssociatedGroups.Contains(name);
}

public static bool HasGroupAssociation(this SPWeb web, SPGroup group)
{
    if (group == null)
        throw new ArgumentNullException("group");
    return web.HasGroupAssociation(group.Name);
}

public static void EnsureGroupAssociation(this SPWeb web, SPGroup group)
{
    if (web.HasGroupAssociation(group))
        web.AssociatedGroups.Add(group);
}

The code should be pretty self-explanatory. The name comparison logic in NameEquals() is written to align with how SharePoint compares group names internally, though they use their own implementation of case insensitivity because the framework's isn't good enough. Or something like that.

There should be two lessons here:

Don't assume methods that have a notion of equality, like Contains(), will behave like you expect.
Use SPExLib and contribute other extensions and helpers you find useful. :)

Extension Methods on Types You Own?

2010-01-25T11:00:00Z

It's no secret that I'm a fan of using extension methods to make code more concise and expressive. This is particularly handy for enhancing APIs outside of your control, from the base class library to ASP.NET MVC and SharePoint. However, there are certain situations where it might be useful to use extension methods even though you have the option to add those methods to the class or interface itself. Consider this simplified caching interface:

public interface ICacheProvider
{
    T Get<T>(string key);
    void Insert<T>(string key, T value);
}

And a simple application of the decorator pattern to implement a cached repository:

public class CachedAwesomeRepository : IAwesomeRepository
{
    private readonly IAwesomeRepository awesomeRepository;
    private readonly ICacheProvider cacheProvider;

    public CachedAwesomeRepository(IAwesomeRepository awesomeRepository, ICacheProvider cacheProvider)
    {
        this.awesomeRepository = awesomeRepository;
        this.cacheProvider = cacheProvider;
    }

    public Awesome GetAwesome(string id)
    {
        var awesome = cacheProvider.Get<Awesome>(id);
        if(awesome == null)
            cacheProvider.Insert(id, (awesome = awesomeRepository.GetAwesome(id)));
        return awesome;
    }
}

So far, so good. However, as caching is used more often it becomes clear that there's a common pattern that we might want to extract:

    T ICacheProvider.GetOrInsert<T>(string key, Func<T> valueFactory)
    {
        T value = Get<T>(key);
        if(value == default(T))
            Insert(key, (value = valueFactory()));
        return value;
    }

Which would reduce GetAwesome() to a single, simple expression:

    public Awesome GetAwesome(string id)
    {
        return cacheProvider.GetOrInsert(id, () => awesomeRepository.GetAwesome(id));
    }

Now I just need to decide where GetOrInsert() lives. Since I control ICacheProvider, I could just add another method to the interface and update all its implementers. However, after starting down this path, I concluded this was not desirable for a number of reasons:

The implementation of GetOrInsert within each cache provider was essentially identical.
Tests using a mocked ICacheProvider now needed to know if the code under test used GetOrInsert() or Get() + Insert(), coupling the test too tightly to the implementation. Furthermore, natural tests along the lines of "Should return cached value" and "Should insert value from repository if not cached" were replaced with a single implementation-specific test: "Should return value from GetOrInsert".
Most importantly, I came to realize that GetOrInsert() really just isn't something that a cache does, so why should it be part of the interface?

So instead I have a handy GetOrInsert() extension method (conversion is left as an exercise for the reader) that I can use to clean up my caching code without needing to change any of my cache providers or tests for existing consumers.

The question is really analogous to whether or not Select() and Where() should be part of IEnumerable<T>. They are certainly useful ways to consume the interface, just as GetOrInsert() is, but they exist outside of what an IEnumerable<T> really is.

Quick Tip: Parse String to Nullable Value

2010-01-23T16:39:00Z

When you're dealing with a system like SharePoint that returns most data as strings, it's common to want to parse the data back into a useful numeric format. The .NET framework offers several options to achieve this, namely the static methods on System.Convert and the static Parse() methods on the various value types. However, these are limited in that they turn null string values into the default for the given type (0, false, etc) and they throw exceptions to indicate failure, which might be a performance concern.

Often, a better option is to use the static TryParse() method provided by most value types (with the notable exception of enumerations). These follow the common pattern of returning a boolean to indicate success and using an out parameter to return the value. While much better suited for what we're trying to achieve, the TryParse pattern requires more plumbing than I care to see most of the time—I just want the value. To that end, I put together a simple extension method to encapsulate the pattern:

public delegate bool TryParser<T>(string value, out T result) where T : struct;

public static T? ParseWith<T>(this string value, TryParser<T> parser) where T : struct
{
    T result;
    if (parser(value, out result))
        return result;
    return null;
}

The struct constraint on T is required to align with the constraint on the Nullable<T> returned by the method.

We can now greatly simplify our efforts to parse nullable values:

var myIntPropStr = properties.BeforeProperties["MyIntProp"] as string;
var myIntProp = myPropStr.ParseWith<int>(int.TryParse);
if(myIntProp == null)
    throw new Exception("MyIntProp is empty!");

One quirk of this technique is that Visual Studio usually cannot infer T from just the TryParse method because of its multiple overloads. One option would be to write a dedicated method for each value type, but I would view this as unnecessary cluttering of the string type. Your mileage may vary.

Selecting Static Results with Dynamic LINQ

2010-01-23T07:05:00Z

Dynamic LINQ (DLINQ) is a LINQ extension provided in the VS 2008 Samples. Scott Guthrie provides a good overview here: Dynamic LINQ (Part 1: Using the LINQ Dynamic Query Library), but the executive summary is that it implements certain query operations on IQueryable (the non-generic variety), with filtering, grouping and projection specified with strings rather than statically-typed expressions.

I've never had a use for it, but a question on Stack Overflow caused me to take a second look...

...the selected groupbyvalue (Group) will always be a string, and the sum will always be a double, so I want to be able to cast into something like a List, where Result is an object with properties Group (string) and TotalValue (double).

Before we can solve the problem, let's take a closer look at why it is being asked...

DynamicExpression.CreateClass

We can use the simplest of dynamic queries to explore a bit:

[Test]
public void DLINQ_IdentityProjection_ReturnsDynamicClass()
{
    IQueryable nums = Enumerable.Range(1, 5).AsQueryable();
    IQueryable q = nums.Select("new (it as Value)");
    Type elementType = q.ElementType;

    Assert.AreEqual("DynamicClass1", elementType.Name);
    CollectionAssert.AreEqual(new[] { typeof(int) },
        elementType.GetProperties().Select(p => p.PropertyType).ToArray());
}

DLINQ defines a special expression syntax for projection that is used to specify what values should be returned and how. it refers to the current element, which in our case is an int.

The result in question comes from DynamicQueryable.Select():

public static IQueryable Select(this IQueryable source, string selector, params object[] values)
{
    LambdaExpression lambda = DynamicExpression.ParseLambda(source.ElementType, null, selector, values);
    return source.Provider.CreateQuery(
        Expression.Call(
            typeof(Queryable), "Select",
            new Type[] { source.ElementType, lambda.Body.Type },
            source.Expression, Expression.Quote(lambda)));
}

The non-generic return type suggest that the type of the values returned is unknown at compile time. If we check an element's type at runtime, we'll see something like DynamicClass1. Tracing down the stack from DynamicExpression.ParseLambda(), we eventually find that DynamicClass1 is generated by a call to DynamicExpression.CreateClass() in ExpressionParser.ParseNew(). CreateClass() in turn delegates to a static ClassFactory which manages a dynamic assembly in the current AppDomain to hold the new classes, each generated by Reflection.Emit. The resulting type is then used to generate the MemberInit expression that constructs the object.

Dynamic to Static

While dynamic objects are useful in some situations (thus support in C# 4), in this case we want to use static typing. Let's specify our result type with a generic method:

IQueryable<TResult> Select<TResult>(this IQueryable source, string selector, params object[] values);

We just need a mechanism to insert our result type into DLINQ to supersede the dynamic result. This is surprisingly easy to implement, as ParseLambda() already accepts a resultType argument. We just need to capture it...

private Type resultType;
public Expression Parse(Type resultType)
{
    this.resultType = resultType;
    int exprPos = token.pos;
    // ...

...and then update ParseNew() to use the specified type:

Expression ParseNew()
{
    // ...
    NextToken();
    Type type = this.resultType ?? DynamicExpression.CreateClass(properties);
    MemberBinding[] bindings = new MemberBinding[properties.Count];
    for (int i = 0; i < bindings.Length; i++)
        bindings = Expression.Bind(type.GetProperty(properties.Name), expressions);
    return Expression.MemberInit(Expression.New(type), bindings);
}

If resultType is null, as it is in the existing Select() implementation, a DynamicClass is used instead.

The generic Select<TResult> is then completed by referencing TResult as appropriate:

public static IQueryable<TResult> Select<TResult>(this IQueryable source, string selector, params object[] values)
{
    LambdaExpression lambda = DynamicExpression.ParseLambda(source.ElementType, typeof(TResult), selector, values);
    return source.Provider.CreateQuery<TResult>(
        Expression.Call(
            typeof(Queryable), "Select",
            new Type[] { source.ElementType, typeof(TResult) },
            source.Expression, Expression.Quote(lambda)));
}

With the following usage:

public class ValueClass { public int Value { get; set; } }

[Test]
public void DLINQ_IdentityProjection_ReturnsStaticClass()
{
    IQueryable nums = Enumerable.Range(1, 5).AsQueryable();
    IQueryable<ValueClass> q = nums.Select<ValueClass>("new (it as Value)");
    Type elementType = q.ElementType;

    Assert.AreEqual("ValueClass", elementType.Name);
    CollectionAssert.AreEqual(nums.ToArray(), q.Select(v => v.Value).ToArray());
}

Note that the property names in TResult must match those in the Select query exactly. Changing the query to "new (it as value)" results in an unhandled ArgumentNullException in the Expression.Bind() call seen in the for loop of ParseNew() above, as the "value" property cannot be found.

Selecting Anonymous Types

So we can select dynamic types or existing named types, but what if we want to have the benefits of static typing without having to declare a dedicated ValueClass, as we can with anonymous types and normal static LINQ? As a variation on techniques used elsewhere, let's can define an overload of Select() that accepts an instance of the anonymous type whose values we will ignore but using its type to infer the desired return type. The overload is trivial:

public static IQueryable<TResult> Select<TResult>(this IQueryable source, TResult template, string selector, params object[] values)
{
    return source.Select<TResult>(selector, values);
}

With usage looking like this (note the required switch to var q):

[Test]
public void DLINQ_IdentityProjection_ReturnsStaticClass()
{
    IQueryable nums = Enumerable.Range(1, 5).AsQueryable();
    var q = nums.Select(new { Value = 0 }, "new (it as Value)");
    Type elementType = q.ElementType;

    Assert.IsTrue(elementType.Name.Contains("AnonymousType"));
    CollectionAssert.AreEqual(nums.ToArray(), q.Select(v => v.Value).ToArray());
}

However, if we try the above we encounter an unfortunate error:

The property 'Int32 Value' has no 'set' accessor

As you may or may not know, anonymous types in C# are immutable (modulo changes to objects they reference), with their values set through a compiler-generated constructor. (I'm not sure if this is true in VB.) With this knowledge in hand, we can update ParseNew() to check if resultType has such a constructor that we could use instead:

    // ...
    Type type = this.resultType ?? DynamicExpression.CreateClass(properties);

    var propertyTypes = type.GetProperties().Select(p => p.PropertyType).ToArray();
    var ctor = type.GetConstructor(propertyTypes);
    if (ctor != null)
        return Expression.New(ctor, expressions);

    MemberBinding[] bindings = new MemberBinding[properties.Count];
    for (int i = 0; i < bindings.Length; i++)
        bindings = Expression.Bind(type.GetProperty(properties.Name), expressions);
    return Expression.MemberInit(Expression.New(type), bindings);
}

And with that we can now project from a dynamic query onto static types, both named and anonymous, with a reasonably natural interface.

Due to licensing I can't post the full example, but if you're at all curious about Reflection.Emit or how DLINQ works I would encourage you to dive in and let us know what else you come up with. Things will get even more interesting with the combination of LINQ, the DLR and C# 4's dynamic in the coming months.

HTTP Error Codes in WatiN 1.3

2010-01-12T05:11:00Z

One of the biggest surprises when I started working with WatiN was the omission of a mechanism to check for error conditions. A partial solution using a subclass has been posted before, but it doesn't quite cover all the bases. Specifically, it's missing a mechanism to attach existing Internet Explorer instances to objects of the enhanced subtype. Depending on the site under test's use of pop-ups, this could be a rather severe limitation. So let's see how we can fix it.

As WatiN is open source, one option is to just patch the existing implementation to include the desired behavior. I've uploaded a patch with tests here, but the gist of the patch is quite similar to the solution referenced above:

protected void AttachEventHandlers()
{
    ie.BeforeNavigate2 += (object pDisp, ref object URL, ref object Flags, ref object TargetFrameName, ref object PostData, ref object Headers, ref bool Cancel) =>
    {
        ErrorCode = null;
    };
    ie.NavigateError += (object pDisp, ref object URL, ref object Frame, ref object StatusCode, ref bool Cancel) =>
    {
        ErrorCode = (HttpStatusCode)StatusCode;
    };
}

/// <summary>
/// HTTP Status Code of last error, or null if the last request was successful
/// </summary>
public HttpStatusCode? ErrorCode
{
    get;
    private set;
}

Before every request we clear out the error code, with errors captured as an enum value borrowed from System.Net.

We complete the patch by placing calls to our AttachEventHandlers() method in two places:

The constructor that accepts an existing SHDocVw.InternetExplorer handle.
The CreateNewIEAndGoToUri() method used by every other constructor.

At this point we can now assert success:

using (IE ie = new IE("http://solutionizing.net/"))
{
    Assert.That(ie.ErrorCode, Is.Null);
}

Or specific kinds of failure:

using (IE ie = new IE("http://solutionizing.net/4040404040404"))
{
    Assert.That(ie.ErrorCode, Is.EqualTo(HttpStatusCode.NotFound));
}

See the patch above for a more complete set of example tests.

Private Strikes Again

It's wonderful that we have the option to make our own patched build with the desired behavior, but what if we would rather use the binary distribution? Well through the magic of inheritance we can get most of the way there pretty easily:

public class MyIE : IE
{
    public MyIE()
    {
        Initialize();
    }
    public MyIE(object shDocVwInternetExplorer)
        : base(shDocVwInternetExplorer)
    {
        Initialize();
    }
    public MyIE(string url)
        : base(url)
    {
        Initialize();
    }

    // Remaining c'tors left as an exercise

    // Property named ie for consistency with the private field in the parent
    protected InternetExplorer ie
    {
        get { return (InternetExplorer)InternetExplorer; }
    }

    protected void Initialize()
    {
        AttachEventHandlers();
    }

    // AttachEventHandlers() and ErrorCode as defined above
}

But as I suggested before, this is where we run into a bit of a snag. The IE class also provides a set of static AttachToIE() methods that, as their name suggests, return an IE object for an existing Internet Explorer window. These static methods have the downside that they are hard-coded to return objects of type IE, not our enhanced MyIE type. And because all the relevant helper methods are private and not designed for reuse, we have no choice but to pull them into our subclass in their entirety:

    public new static MyIE AttachToIE(BaseConstraint findBy)
    {
        return findIE(findBy, Settings.AttachToIETimeOut, true);
    }
    public new static MyIE AttachToIE(BaseConstraint findBy, int timeout)
    {
        return findIE(findBy, timeout, true);
    }
    public new static MyIE AttachToIENoWait(BaseConstraint findBy)
    {
        return findIE(findBy, Settings.AttachToIETimeOut, false);
    }
    public new static MyIE AttachToIENoWait(BaseConstraint findBy, int timeout)
    {
        return findIE(findBy, timeout, false);
    }

    private static MyIE findIE(BaseConstraint findBy, int timeout, bool waitForComplete)
    {
        SHDocVw.InternetExplorer internetExplorer = findInternetExplorer(findBy, timeout);

        if (internetExplorer != null)
        {
            MyIE ie = new MyIE(internetExplorer);
            if (waitForComplete)
            {
                ie.WaitForComplete();
            }

            return ie;
        }

        throw new IENotFoundException(findBy.ConstraintToString(), timeout);
    }

    protected static SHDocVw.InternetExplorer findInternetExplorer(BaseConstraint findBy, int timeout)
    {
        Logger.LogAction("Busy finding Internet Explorer matching constriant " + findBy.ConstraintToString());

        SimpleTimer timeoutTimer = new SimpleTimer(timeout);

        do
        {
            Thread.Sleep(500);

            SHDocVw.InternetExplorer internetExplorer = findInternetExplorer(findBy);

            if (internetExplorer != null)
            {
                return internetExplorer;
            }
        } while (!timeoutTimer.Elapsed);

        return null;
    }

    private static SHDocVw.InternetExplorer findInternetExplorer(BaseConstraint findBy)
    {
        ShellWindows allBrowsers = new ShellWindows();

        int browserCount = allBrowsers.Count;
        int browserCounter = 0;

        IEAttributeBag attributeBag = new IEAttributeBag();

        while (browserCounter < browserCount)
        {
            attributeBag.InternetExplorer = (SHDocVw.InternetExplorer) allBrowsers.Item(browserCounter);

            if (findBy.Compare(attributeBag))
            {
                return attributeBag.InternetExplorer;
            }

            browserCounter++;
        }

        return null;
    }

The original version of the first findInternetExplorer() is private. Were it protected instead, we would only have had to implement our own findIE() to wrap the found InternetExplorer object in our subtype.

I won't go so far as to say private methods are a code smell, but they certainly can make the O in OCP more difficult to achieve.

So there you have it, two different techniques for accessing HTTP error codes in WatiN 1.3. At some point I'll look at adding similar functionality to 2.0, if it's not already there. And if someone on the project team see this, feel free to run with it.

Script to Enable HTTP Compression (Gzip/Deflate) in IIS 6

2010-01-08T05:09:00Z

One of the easiest ways to improve web site performance is to enable HTTP compression (often referred to as GZIP compression), which trades CPU time to compress content for a reduced payload delivered over the wire. In the vast majority of cases, the trade-off is a good one. When implementing HTTP compression, your content will break down into three categories:

Content that should not be compressed because it is already compressed: images, PDF files, audio, video, etc.
Static content that can be compressed once and cached for later.
Dynamic content that needs to be compressed for every request.

Excluding already-compressed content will need to be considered regardless of the techniques used to compress categories 2 and 3. Since version 5, IIS has included support for both kinds of HTTP compression. This can be enabled through the management interface, but you will almost certainly want to tweak the default configuration in the metabase (see script below). While IIS works great for compressing static files, its extension-based configuration is rather limited when serving up dynamic content, especially if you don't use extensions (as with most ASP.NET MVC routes) or you serve dynamic content that should not be compressed. A better solution is provided in HttpCompress by Ben Lowery, a configurable HttpModule that allows content to be excluded from compression by MIME type. A standard configuration might look something like this:

<configuration>
  ...
  <blowery.web>
    <httpCompress preferredAlgorithm="gzip" compressionLevel="normal">
      <excludedMimeTypes>
        <add type="image/jpeg" />
        <add type="image/png" />
        <add type="image/gif" />
        <add type="application/pdf" />
      </excludedMimeTypes>
      <excludedPaths></excludedPaths>
    </httpCompress>
  </blowery.web>
  ...
</configuration>

To supplement the compressed dynamic content, you should also enable static compression for the rest of your not-already-compressed content. The script should be pretty self-explanatory, but I'll draw attention to a few things:

The tcfpath variable at the top is currently set to IIS's default location, which you are free to change.
The extlist variable accepts a space-delimited list of file extensions that should be compressed. Again, only include files types that are not already compressed, as recompressing a file wastes cycles and can actually make some files larger.
There are a few other metabase properties that can also be set, including compression level, but these are the bare minimum.
I have been told repeatedly that IISRESET should be sufficient to apply the metabase changes, but I could not get it to work as consistently as manually restarting the IIS Admin Service — YMMV.
If all goes well, the nice arrow at the end will point to True.

If you have anything else to add, or have problems with the script, please let me know.

@echo off
set adsutil=C:\Inetpub\AdminScripts\adsutil.vbs
set tcfpath=%windir%\IIS Temporary Compressed Files
set extlist=css htm html js txt xml

mkdir "%tcfpath%"

echo Ensure IIS_WPG has Full Control on %tcfpath%

explorer "%tcfpath%\.."
pause

cscript.exe %adsutil% set w3svc/Filters/Compression/Parameters/HcDoStaticCompression true 
cscript.exe %adsutil% set w3svc/Filters/Compression/Parameters/HcCompressionDirectory "%tcfpath%"
cscript.exe %adsutil% set w3svc/Filters/Compression/DEFLATE/HcFileExtensions %extlist%
cscript.exe %adsutil% set w3svc/Filters/Compression/GZIP/HcFileExtensions %extlist%

echo Restart IIS Admin Service - IISRESET does not seem to work
pause

echo Close Services to continue...
Services.msc

cscript.exe %adsutil% get w3svc/Filters/Compression/Parameters/HcDoStaticCompression

echo Should be True -----------------------------^^

pause

Is Functional Abstraction Too Clever?

2009-10-17T07:29:00Z

I received a rather interesting comment on a recent Stack Overflow answer:

This code seems too clever by half. Is it art? – PeterAllenWebb

The code in question was a functional solution to an algorithm described approximately as follows:

Draw n−1 numbers at random, in the range 1 to m−1. Add 0 and m to the list and order these numbers. The difference between each two consecutive numbers gives you a return value.

Which I solved like this, with n = slots and m = max:

static int[] GetSlots(int slots, int max)
{
    return new Random().Values(1, max)
                       .Take(slots - 1)
                       .Append(0, max)
                       .OrderBy(i => i)
                       .Pairwise((x, y) => y - x)
                       .ToArray();
}

Using a few extension methods:

Values() returns an infinite sequence of random values within the specified range.
Append() takes a params array and appends its arguments to the original sequence.
Pairwise() generates a sequence from calculations on pairs of consecutive elements in the original sequence.

I can see how one would think the code is clever; however, I'm not sure what would qualify it as too clever. Every method call has a well-defined purpose and maps directly to part of the original algorithm:

From random numbers in the range 1 to m−1...
...draw n−1.
Add 0 and m to the list...
...and order these numbers.
The difference between each two consecutive numbers...
...gives you a return value [in the array].

As far as I'm concerned, a solution couldn't get much clearer than this, but that's easy enough for me to say—what do you think? Is there a better way to express the algorithm? Would an imperative solution with shared state be more readable? How about maintainable?

For example, one could add the requirement that the random numbers not be repeated so that the difference between adjacent numbers is always nonzero. Updating the functional solution is as simple as adding a Distinct() call:

    return new Random().Values(1, max)
                       .Distinct()
                       .Take(slots - 1)
                       ...

To me, this is the value proposition of functional programming. By expressing the algorithm in terms of common operations, we're able to spend more time thinking about the problem than the details of the solution. A similar change in an imperative implementation would almost certainly have been more involved and prone to error.

For completeness, here are the implementations of the extension methods used:

public static IEnumerable<int> Values(this Random random, int minValue, int maxValue)
{
    while (true)
        yield return random.Next(minValue, maxValue);
}

public static IEnumerable<TResult> Pairwise<TSource, TResult>(
    this IEnumerable<TSource> source,
    Func<TSource, TSource, TResult> resultSelector)
{
    TSource previous = default(TSource);

    using (var it = source.GetEnumerator())
    {
        if (it.MoveNext())
            previous = it.Current;

        while (it.MoveNext())
            yield return resultSelector(previous, previous = it.Current);
    }
}

public static IEnumerable<T> Append<T>(this IEnumerable<T> source, params T[] args)
{
    return source.Concat(args);
}

This also reminds me that Jimmy posted a similar Append() method as part of his latest post on missing LINQ operators. I used to use a version similar to his, but have found the params version to be more flexible (and easier to implement). Its Prepend() counterpart is similarly trivial:

public static IEnumerable<T> Prepend<T>(this IEnumerable<T> source, params T[] args)
{
    return args.Concat(source);
}

Refactoring with Iterators: Prime Factors

2009-09-30T15:17:00Z

Andrew Woodward recently posted a comparison of his test-driven Prime Factors solution to one written by Uncle Bob. In the comments, someone suggested that Andrew use an iterator instead so I thought I'd give it a try.

First, let's repost the original code:

private const int SMALLEST_PRIME = 2;

public List<int> Generate(int i)
{
    List<int> primes = new List<int>();
    int divider = SMALLEST_PRIME;
    while (HasPrimes(i))
    {
        while (IsDivisable(i, divider))
        {
            i = AddPrimeToProductsAndReduce(i, primes, divider);
        }
        divider++;
    }
    return primes;
}

private bool IsDivisable(int i, int divider)
{
    return i%divider == 0;
}

private bool HasPrimes(int i)
{
    return i >= SMALLEST_PRIME;
}

private int AddPrimeToProductsAndReduce(int i, List<int> primes, int prime)
{
    primes.Add(prime);
    i /= prime;
    return i;
}

By switching our method to return IEnumerable<int>, we can replace the primes list with an iterator. We will also remove the AddPrimeToProducts functionality from that helper method since we don't have the list any more:

public IEnumerable<int> Generate(int i)
{
    int divider = SMALLEST_PRIME;
    while (HasPrimes(i))
    {
        while (IsDivisable(i, divider))
        {
            yield return divider;
            i = Reduce(i, divider);
        }
        divider++;
    }
}

private int Reduce(int i, int prime)
{
    return i / prime;
}

I think this is a good change for three reasons:

There's nothing about the problem that requires a List<int> be returned, we just want a sequence of the factors.
AddPrimeToProductsAndReduce suggested that it had a side effect, but exactly what wasn't immediately obvious.
It's much easier to see what values are being included in the result.

That said, I think we can clean this up even more with a second iterator. Specifically, I think we should break out the logic for our candidate factors:

private IEnumerable<int> Divisors
{
    get
    {
        int x = SMALLEST_PRIME;
        while (true)
            yield return x++;
    }
}

Which allows us to separate the logic for generating a divider from the code that consumes it:

public IEnumerable<int> Generate(int toFactor)
{
    foreach (var divider in Divisors)
    {
        if (!HasPrimes(toFactor))
            break;

        while (IsDivisable(toFactor, divider))
        {
            yield return divider;
            toFactor = Reduce(toFactor, divider);
        }
    }
}

We should also eliminate the negation by flipping HasPrimes to become IsFactored:

public IEnumerable<int> Generate(int toFactor)
{
    foreach (var divider in Divisors)
    {
        if (IsFactored(toFactor))
            break;

        while (IsDivisable(toFactor, divider))
        {
            yield return divider;
            toFactor = Reduce(toFactor, divider);
        }
    }
}

private bool IsFactored(int i)
{
    return i <= 1;
}

This does introduce a (very) minor inefficiency in that the Divisors enumerator will MoveNext() one extra time before breaking out of the loop, which could be mitigated by checking IsFactored both before the foreach and after the while loop. Less readable, insignificantly more efficient...take your pick.

The other advantage to breaking out the logic to generate Divisors is that we can easily pick smarter candidates. One option is to skip even numbers greater than 2. An even better optimization takes advantage of the fact that all primes greater than 3 are of the form x±1 where x is a multiple of 6:

private IEnumerable<int> Divisors
{
    get
    {
        yield return 2;
        yield return 3;
        int i = 6;
        while (true)
        {
            yield return i - 1;
            yield return i + 1;
            i += 6;
        }
    }
}

Implementing this sort of logic in the original version would have been much more difficult, both in terms of correctness and readability.

Hacking LINQ Expressions: Join With Comparer

2009-09-19T18:15:00Z

In this installment of my Hacking LINQ series we'll take a look at providing an IEqualityComparer for use in a LINQ join clause.

The Problem

Many of the Standard Query Operators require comparing sequence elements and the default query providers are kind enough to give us overloads that accept a suitable comparer. Among these operators, Join and GroupJoin have perhaps the most useful query syntax:

 var res = from s in States
          join a in AreaCodes
            on s.Abbr equals a.StateAbbr
          select new { s.Name, a.AreaCode };

While a bit more verbose, I find the intent much easier to read then the method equivalent:

var res = States.Join(AreaCodes,
                      s => s.Abbr, a => a.StateAbbr,
                      (s, a) => new { s.Name, a.AreaCode });

Or maybe I've just spent too much time in SQL. Either way, I thought it would be useful to support joins by a comparer.

The Goal

We will use another extension method to specify how the join should be performed:

var res = from s in States
          join a in AreaCodes.WithComparer(StringComparer.OrdinalIgnoreCase)
            on s.Abbr equals a.StateAbbr
          select new { s.Name, a.AreaCode };

We can also support the same syntax for group joins:

var res = from s in States
          join a in AreaCodes.WithComparer(StringComparer.OrdinalIgnoreCase)
            on s.Abbr equals a.StateAbbr into j
          select new { s.Name, Count = j.Count() };

The Hack

As with most LINQ hacks, we're going to use the result of WithComparer to call a specialized version of Join or GroupJoin, in this case by providing a replacement for the join's inner sequence:

var res = States.Join(AreaCodes.WithComparer(StringComparer.OrdinalIgnoreCase),
                      s => s.Abbr, a => a.StateAbbr,
                      (s, a) => new { s.Name, a.AreaCode });

Eventually leading to this method call:

var res = States.Join(AreaCodes,
                      s => s.Abbr, a => a.StateAbbr,
                      (s, a) => new { s.Name, a.AreaCode },
                      StringComparer.OrdinalIgnoreCase);

Since we need both the inner collection we're extending and the comparer, we can guess our extension method will be implemented something like this:

public static JoinComparerProvider<T, TKey> WithComparer<T, TKey>(
    this IEnumerable<T> inner, IEqualityComparer<TKey> comparer)
{
    return new JoinComparerProvider<T, TKey>(inner, comparer);
}

With a trivial provider implementation:

public sealed class JoinComparerProvider<T, TKey>
{
    internal JoinComparerProvider(IEnumerable<T> inner, IEqualityComparer<TKey> comparer)
    {
        Inner = inner;
        Comparer = comparer;
    }

    public IEqualityComparer<TKey> Comparer { get; private set; }
    public IEnumerable<T> Inner { get; private set; }
}

The final piece is our Join overload:

public static IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(
    this IEnumerable<TOuter> outer,
    JoinComparerProvider<TInner, TKey> inner,
    Func<TOuter, TKey> outerKeySelector,
    Func<TInner, TKey> innerKeySelector,
    Func<TOuter, TInner, TResult> resultSelector)
{
    return outer.Join(inner.Inner, outerKeySelector, innerKeySelector,
                      resultSelector, inner.Comparer);
}

Implementations of GroupJoin and their IQueryable counterparts are similarly trivial.

Hacking LINQ Expressions: Select With Index

2009-09-15T09:04:00Z

First, a point of clarification: I use LINQ Expressions to mean (Language-INtegrated) Query Expressions (the language feature) rather than Expression Trees (the .NET 3.5 library in System.Linq.Expressions).

So what do I mean by "Hacking LINQ Expressions"? Quite simply, I'm not content with the rather limited set of operations that query expressions allow me to represent. By understanding how queries are translated, we can use various techniques to broaden our expressive reach. I have already documented one such hack for managing IDisposable objects with LINQ, so I guess we can call this the second in an unbounded series.

The Problem

In thinking over use cases for functional construction of web control trees, I paused to think through how I would express alternate row styling. My mind immediately jumped to the overload of Select() that exposes the current element's index:

Controls.Add(
    new Table().WithControls(
        data.Select((x, i) =>
            new TableRow() {
                CssClass = i % 2 == 0 ? "" : "alt"
            }.WithControls(
                new TableCell().WithControls(x)
            )
        )
    )
);

This works fine for simple cases, but breaks down for more complex queries:

Controls.Add(
    new Table().WithControls((
        from x in Xs
        join y in Ys on x.Key equals y.Key
        select new { x, y }
        ).Select((z, i) =>
            new TableRow() {
                CssClass = i % 2 == 0 ? "" : "alt"
            }.WithControls(
                new TableCell().WithControls(z.x.ValueX, z.y.ValueY)
            )
        )
    )
);

The Goal

Instead, I propose a simple extension method to retrieve an index at arbitrary points in a query:

var res = from x in data
          from i in x.GetIndex()
          select new { x, i };

Or our control examples:

Controls.Add(
    new Table().WithControls(
        from x in data
        from i in x.GetIndex()
        select new TableRow() {
            CssClass = i % 2 == 0 ? "" : "alt"
        }.WithControls(
            new TableCell().WithControls(x)
        )
    )
);

Controls.Add(
    new Table().WithControls(
        from x in Xs
        join y in Ys on x.Key equals y.Key
        from i in y.GetIndex()
        select new TableRow() {
            CssClass = i % 2 == 0 ? "" : "alt"
        }.WithControls(
            new TableCell().WithControls(x.ValueX, y.ValueY)
        )
    )
);

Much like in the IDisposable solution, we use a from clause to act as an intermediate assignment. But in this case our hack is a bit trickier than a simple iterator.

The Hack

For this solution we're going to take advantage of how multiple from clauses are translated:

var res = data.SelectMany(x => x.GetIndex(), (x, i) => new { x, i });

Looking at the parameter list, we see that our collectionSelector should return the result of x.GetIndex() and our resultSelector's second argument needs to be an int:

public static IEnumerable<TResult> SelectMany<TSource, TResult>(
    this IEnumerable<TSource> source,
    Func<TSource, SelectIndexProvider> collectionSelector,
    Func<TSource, int, TResult> resultSelector)

The astute observer will notice that the signature of this resultSelector exactly matches the selector used by Select's with-index overload, trivializing the method implementation:

{
    return source.Select(resultSelector);
}

Note that we're not even using collectionSelector! We're just using its return type as a flag to force the compiler to use this version of SelectMany(). The rest of the pieces are incredibly simple now that we know the actual SelectIndexProvider value is never used:

public sealed class SelectIndexProvider
{
    private SelectIndexProvider() { }
}

public static SelectIndexProvider GetIndex<T>(this T element)
{
    return null;
}

And for good measure, an equivalent version to extend IQueryable<>:

public static IQueryable<TResult> SelectMany<TSource, TResult>(
    this IQueryable<TSource> source,
    Expression<Func<TSource, SelectIndexProvider>> collectionSelector,
    Expression<Func<TSource, int, TResult>> resultSelector)
{
    return source.Select(resultSelector);
}

Because we're just calling Select(), the query expression isn't even aware of the call to GetIndex():

System.Linq.Enumerable+<RangeIterator>d__b1.Select((x, i) => (x * i))

We're essentially providing our own syntactic sugar over the sugar already provided by query expressions. Pretty sweet, eh?

As a final exercise for the reader, what would this print?

var res = from x in Enumerable.Range(1, 5)
          from i in x.GetIndex()
          from y in Enumerable.Repeat(i, x)
          where y % 2 == 1
          from j in 0.GetIndex()
          select i+j;

foreach (var r in res)
    Console.WriteLine(r);

Functional Construction for ASP.NET Web Forms

2009-09-13T08:59:00Z

System.Xml.Linq (a.k.a. LINQ to XML) introduces a nifty approach to creating XML elements called functional construction. I'm not entirely sure why they call it functional given that constructing an object graph is a decidedly non-functional task in the traditional sense of the word, but I digress.

Functional construction has three key features:

Constructors accept arguments of various types, handling them appropriately.
Constructors accept a params array of type Object to enable creation of complex objects.
If an argument implements IEnumerable, the objects within the sequence are added.

If you haven't seen it in action, I encourage you to take a look at the examples on MSDN and elsewhere—it really is pretty slick. This post will show how a similar technique can be used to build control trees in ASP.NET web forms (and probably WinForms with minimal adjustment).

Basic functional construction can be implemented using two relatively simple extension methods:

public static void Add(this ControlCollection @this, object content)
{
    if (content is Control)
        @this.Add((Control)content);
    else if (content is IEnumerable)
        foreach (object c in (IEnumerable)content)
            @this.Add(c);
    else if (content != null)
        @this.Add(new LiteralControl(content.ToString()));
}

public static void Add(this ControlCollection @this, params object[] args)
{
    @this.Add((IEnumerable)args);
}

We handle four cases:

Control? Add it.
Sequence? Add each.
Other value? Add literal.
Null? Ignore.

And our params overload just calls its arguments a sequence and defers to the other.

In the time-honored tradition of contrived examples:

Controls.Add(
    new Label() { Text = "Nums:" },
    "&nbsp;",
    from i in Enumerable.Range(1, 6)
    group i by i % 2
);

This would render "Nums: 135246". Note that the result of that LINQ expression is a sequence of sequences, which is flattened automatically and converted into literals. For comparison, here's an equivalent set of statements:

Controls.Add(new Label() { Text = "Nums:" });
Controls.Add(new LiteralControl("&nbsp;"));
foreach (var g in from i in Enumerable.Range(1, 6)
                  group i by i % 2)
    foreach (var i in g)
        Controls.Add(new LiteralControl(i.ToString()));

Hopefully seeing them side by side makes it clear why this new method of construction might have merit. But we're not done yet.

Expressions, Expressions, Expressions

Many language features introduced in C# 3.0 and Visual Basic 9 make expressions increasingly important. By expressions I mean a single "line" of code that returns a value. For example, an object initializer is a single expression...

var tb = new TextBox()
{
    ID = "textBox1",
    Text = "Text"
};

... that represents several statements ...

var tb = new TextBox()
tb.ID = "textBox1";
tb.Text = "Text";

That single TextBox expression can then be used in a number of places that its statement equivalent can't: in another object initializer, in a collection initializer, as a parameter to a method, in a .NET 3.5 expression tree, the list goes on. Unfortunately, many older APIs simply aren't built to work in an expression-based world. In particular, initializing subcollections is a considerable pain. However, we can extend the API to handle this nicely:

public static T WithControls<T>(this T @this, params object[] content) where T : Control
{
    if(@this != null)
        @this.Controls.Add(content);
    return @this;
}

The key is the return value: Control in, Control out. We can now construct and populate a container control with a single expression. For example, we could build a dictionary list (remember those?) from our groups:

Controls.Add(
    new HtmlGenericControl("dl")
    .WithControls(
        from i in Enumerable.Range(1, 6)
        group i by i % 2 into g
        select new [] {
            new HtmlGenericControl("dt")
            { InnerText = g.Key == 0 ? "Even" : "Odd" },
            new HtmlGenericControl("dd")
            .WithControls(g)
        }
    )
);

Which would render this:

Odd
135
Even
246

Without the ability to add controls within an expression, this result would require nested loops with local variables to store references to the containers. The actual code produced by the compiler would be nearly identical, but I find the expressions much easier to work with. Similarly, we can easily populate tables. Let's build a cell per number:

Controls.Add(
    new Table().WithControls(
        from i in Enumerable.Range(1, 6)
        group i by i % 2 into g
        select new TableRow().WithControls(
            new TableCell()
            { Text = g.Key == 0 ? "Even" : "Odd" },
            g.Select(n => new TableCell().WithControls(n))
        )
    )
);

In a future post I'll look at some other extensions we can use to streamline the construction and initialization of control hierarchies.

Simplifying LazyLinq

2009-09-13T05:09:00Z

This is the fourth in a series of posts on LazyLinq, a wrapper to support lazy initialization and deferred disposal of a LINQ query context:

Introducing LazyLinq: Overview
Introducing LazyLinq: Internals
Introducing LazyLinq: Queryability
Simplifying LazyLinq
Introducing LazyLinq: Lazy DataContext

As I was iterating on the proof of concept for LazyLinq, I always wanted to get rid of the TQuery type parameter. I thought I needed it to distinguish between ordered and unordered wrapped queries, but it just felt messy. The underlying provider mechanism didn’t need it, so why should I?

Well after taking a closer look at the SQL query provider, I figured out how to eliminate it. The object of inspiration is System.Data.Linq.DataQuery<T>, defined as follows:

internal sealed class DataQuery<T> :
    IOrderedQueryable<T>, IQueryable<T>,
    IOrderedQueryable, IQueryable,
    IEnumerable<T>, IEnumerable,
    IQueryProvider,
    IListSource

The key was realizing that IOrderedQueryable<> and ILazyOrderedQueryable<> don’t actually do anything. Implementation-wise, they’re just IQueryable<> or ILazyQueryable<> with an extra interface on top. It’s only on the design side that it actually matters, essentially providing a hook for additional ordering with ThenBy. In LINQ to SQL’s case, that means supporting orderability is as simple as specifying that the query object is both IQueryable<> and IOrderedQueryable<>.

So how does this revelation simplify Lazy LINQ? First, it allows us to remove TQuery from the interfaces:

    public interface ILazyContext<TContext> : IDisposable
    {
        TContext Context { get; }
        ILazyQueryable<TContext, TResult>
            CreateQuery<TResult>(Func<TContext, IQueryable<TResult>> queryBuilder);
        TResult Execute<TResult>(Func<TContext, TResult> action);
    }

    public interface ILazyQueryable<TContext, TSource> : IQueryable<TSource>

    {
        ILazyContext<TContext> Context { get; }
        Func<TContext, IQueryable<TSource>> QueryBuilder { get; }
    }

    public interface ILazyOrderedQueryable<TContext, TSource>
        : ILazyQueryable<TContext, TSource>, IOrderedQueryable<TSource>

    { }

Note that we can also eliminate ILazyContext.CreateOrderedQuery(), instead assuming that CreateQuery() will return something that can be treated as ILazyOrderedQueryable<> as necessary.

For the concrete implementations, we take the cue from DataQuery<T>, letting LazyQueryableImpl implement ILazyOrderedQueryable<> so we can eliminate LazyOrderedQueryableImpl:

    class LazyQueryableImpl<TContext, TSource>
        : ILazyQueryable<TContext, TSource>, ILazyOrderedQueryable<TContext, TSource>
    {
        // Implementation doesn't change
    }

Finally, our sorting query operations will look more like their counterparts in System.Linq.Queryable, casting the result of CreateQuery() to ILazyOrderedQueryable<>. To keep things readable, we’ll split our CreateOrderedQuery<> helper into separate versions for OrderBy and ThenBy. Note how the types of queryOperation map to the usage of OrderBy (unordered to ordered) and ThenBy (ordered to ordered):

        private static ILazyOrderedQueryable<TContext, TResult>
            CreateOrderByQuery<TSource, TContext, TResult>(
                this ILazyQueryable<TContext, TSource> source,
                Func<IQueryable<TSource>, IOrderedQueryable<TResult>> queryOperation
            )
        {
            return (ILazyOrderedQueryable<TContext, TResult>) source.Context.CreateQuery<TResult>(
               context => queryOperation(source.QueryBuilder(context)));
        }

        private static ILazyOrderedQueryable<TContext, TResult>

            CreateThenByQuery<TSource, TContext, TResult>(
                this ILazyQueryable<TContext, TSource> source,
                Func<IOrderedQueryable<TSource>, IOrderedQueryable<TResult>> queryOperation
            )
        {
            return (ILazyOrderedQueryable<TContext, TResult>) source.Context.CreateQuery<TResult>(
               context => queryOperation((IOrderedQueryable<TSource>) source.QueryBuilder(context)));
        }

Removing TQuery from the query operators is left as an exercise for the reader. Or you can just get the source on CodePlex.

A Note on `IOrderedEnumerable<>`

Having taken advantage of how LINQ to IQueryable handles orderability, it’s worth pointing out that LINQ to Objects uses a different approach, specifying new behavior in IOrderedEnumerable<> that is used to support multiple sort criteria:

public interface IOrderedEnumerable<TElement> : IEnumerable<TElement>, IEnumerable
{
    IOrderedEnumerable<TElement>

        CreateOrderedEnumerable<TKey>(
            Func<TElement, TKey> keySelector,
            IComparer<TKey> comparer, bool descending);
}

Introducing LazyLinq: Queryability

2009-08-21T02:31:00Z

This is the third in a series of posts on LazyLinq, a wrapper to support lazy initialization and deferred disposal of a LINQ query context:

Introducing LazyLinq: Overview
Introducing LazyLinq: Internals
Introducing LazyLinq: Queryability
Simplifying LazyLinq
Introducing LazyLinq: Lazy DataContext

Having defined the LazyLinq interfaces and provided concrete implementations, we're left to provide support for the standard query operators.

Learning from Queryable

Before we try to query ILazyQueryable, it's instructive to look at how System.Linq.Queryable works. There are essentially three types of operators on IQueryable<>:

Deferred queries returning IQueryable<>: Select, Where, etc.
Deferred query returning IOrderedQueryable<>: OrderBy, OrderByDescending, ThenBy, ThenByDescending
Everything else: Aggregate, Count, First, etc.

Reflecting Queryable.Select, modulo error checking, we see the following:

public static IQueryable<TResult> Select<TSource, TResult>(
    this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector)
{
    return source.Provider
        .CreateQuery<TResult>(
            Expression.Call(null,
                ((MethodInfo) MethodBase.GetCurrentMethod())
                    .MakeGenericMethod(new Type[] { typeof(TSource), typeof(TResult) }),
                new Expression[] { source.Expression, Expression.Quote(selector) }));
}

The source's Provider is used to construct a new query whose expression includes the call to Select with the given parameters. An ordered query follows a similar pattern, trusting that the query provider will return an IOrderedQueryable<> as appropriate:

public static IOrderedQueryable<TSource> OrderBy<TSource, TKey>(
    this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector)
{
    return (IOrderedQueryable<TSource>) source.Provider
        .CreateQuery<TSource>(
            Expression.Call(null,
                ((MethodInfo) MethodBase.GetCurrentMethod())
                    .MakeGenericMethod(new Type[] { typeof(TSource), typeof(TKey) }),
                new Expression[] { source.Expression, Expression.Quote(keySelector) }));
}

And finally, everything that's not a query is handled by the provider's Execute method:

public static TSource First<TSource>(this IQueryable<TSource> source)
{
    return source.Provider
        .Execute<TSource>(
            Expression.Call(null,
                ((MethodInfo) MethodBase.GetCurrentMethod())
                    .MakeGenericMethod(new Type[] { typeof(TSource) }),
                new Expression[] { source.Expression }));
}

Querying ILazyQueryable

You may have noticed that the above scenarios map rather closely to the methods provided by ILazyContext:

ILazyQueryable<TContext, TResult, TQuery>
    CreateQuery<TResult, TQuery>(Func<TContext, TQuery> queryBuilder)
    where TQuery : IQueryable<TResult>;

ILazyOrderedQueryable<TContext, TResult, TQuery>
    CreateOrderedQuery<TResult, TQuery>(Func<TContext, TQuery> queryBuilder)
    where TQuery : IOrderedQueryable<TResult>;

TResult Execute<TResult>(Func<TContext, TResult> action);

However, rather than expression trees we're pushing around delegates. Execute seems pretty simple, so let's start there:

public static TSource First<TSource, TContext, TQuery>(
        this ILazyQueryable<TContext, TSource, TQuery> source
    ) where TQuery : IQueryable<TSource>
{
    Func<TContext, TResult> action = context => ???;
    return source.Context.Execute(action);
}

So our source has a Context, which knows how to Execute an action from context to some result. To find that result, we need to leverage the other property of source: QueryBuilder. Recalling that QueryBuilder is a function from TContext to TQuery, and that TQuery is an IQueryable<TSource>, we see something on which we can execute Queryable.First:

public static TSource First<TSource, TContext, TQuery>(
        this ILazyQueryable<TContext, TSource, TQuery> source
    ) where TQuery : IQueryable<TSource>
{
    Func<TContext, TResult> action =
        context => source.QueryBuilder(context).First();
    return source.Context.Execute(action);
}

Now seeing as we have dozens of methods to implement like this, it seems an opportune time for a bit of eager refactoring. Recognizing that the only variance is the method call on the IQueryable, let's extract an extension method that does everything else:

private static TResult Execute<TSource, TContext, TResult, TQuery>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        Func<TQuery, TResult> queryOperation
    ) where TQuery : IQueryable<TSource>
{
    return source.Context.Execute(
        context => queryOperation(source.QueryBuilder(context)));
}

From there, additional lazy operators are just a lambda expression away:

public static TSource First<TSource, TContext, TQuery>(
        this ILazyQueryable<TContext, TSource, TQuery> source
    ) where TQuery : IQueryable<TSource>
{
    return source.Execute(q => q.First());
}

public static TAccumulate Aggregate<TContext, TSource, TQuery, TAccumulate>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        TAccumulate seed, Expression<Func<TAccumulate, TSource, TAccumulate>> func
    ) where TQuery : IQueryable<TSource>
{
    return source.Execute(q => q.Aggregate(seed, func));
}

And now having done the hard part (finding an IQueryable), we can translate that understanding to make similar helpers for queries:

private static ILazyQueryable<TContext, TResult, IQueryable<TResult>>
    CreateQuery<TSource, TContext, TQuery, TResult>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        Func<TQuery, IQueryable<TResult>> queryOperation
    ) where TQuery : IQueryable<TSource>
{
    return source.Context.CreateQuery<TResult, IQueryable<TResult>>(
        context => queryOperation(source.QueryBuilder(context)));
}

private static ILazyOrderedQueryable<TContext, TResult, IOrderedQueryable<TResult>>
    CreateOrderedQuery<TSource, TContext, TQuery, TResult>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        Func<TQuery, IOrderedQueryable<TResult>> queryOperation
    ) where TQuery : IQueryable<TSource>
{
    return source.Context.CreateOrderedQuery<TResult, IOrderedQueryable<TResult>>(
        context => queryOperation(source.QueryBuilder(context)));
}

With similarly trivial query operator implementations:

public static ILazyQueryable<TContext, TResult, IQueryable<TResult>>
    Select<TContext, TSource, TQuery, TResult>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        Expression<Func<TSource, TResult>> selector
    ) where TQuery : IQueryable<TSource>
{
    return source.CreateQuery(q => q.Select(selector));
}

public static ILazyOrderedQueryable<TContext, TSource, IOrderedQueryable<TSource>>
    OrderBy<TContext, TSource, TQuery, TKey>(
        this ILazyQueryable<TContext, TSource, TQuery> source,
        Expression<Func<TSource, TKey>> keySelector
    ) where TQuery : IQueryable<TSource>
{
    return source.CreateOrderedQuery(q => q.OrderBy(keySelector));
}

And the end result:

Introducing LazyLinq: Internals

2009-08-18T05:25:00Z

This is the second in a series of posts on LazyLinq, a wrapper to support lazy initialization and deferred disposal of a LINQ query context:

Introducing LazyLinq: Overview
Introducing LazyLinq: Internals
Introducing LazyLinq: Queryability
Simplifying LazyLinq
Introducing LazyLinq: Lazy DataContext

My first post introduced the three interfaces that LazyLinq provides. Next, we get to implement them.

Implementing ILazyQueryable

First, the interface:

    public interface ILazyQueryable<TContext, TSource, TQuery>
        : IQueryable<TSource>
        where TQuery : IQueryable<TSource>
    {
        ILazyContext<TContext> Context { get; }
        Func<TContext, TQuery> QueryBuilder { get; }
    }

We'll start simple with an implicit implementation of the interface and a trivial constructor:

    class LazyQueryableImpl<TContext, TSource, TQuery>
            : ILazyQueryable<TContext, TSource, TQuery>
            where TQuery : IQueryable<TSource>
    {
            public ILazyContext<TContext> Context { get; private set; }
            public Func<TContext, TQuery> QueryBuilder { get; private set; }

            internal LazyQueryableImpl(ILazyContext<TContext> deferredContext, Func<TContext, TQuery> queryBuilder)
            {
                if (deferredContext == null) throw new ArgumentNullException("deferredContext");
                if (queryBuilder == null) throw new ArgumentNullException("queryBuilder");

                Context = deferredContext;
                QueryBuilder = queryBuilder;
            }

Next, a lazy-loaded query built from our lazy context:

            protected TQuery Query
            {
                get
                {
                    if (query == null)
                    {
                        query = QueryBuilder(Context.Context);
                        if (query == null)
                            throw new InvalidOperationException("Query built as null.");
                    }
                    return query;
                }
            }

And the internals of managing Context, which implements IDisposable:

            private void Dispose()
            {
                Context.Dispose();
                query = default(TQuery);
            }

            private IEnumerator<TSource> GetEnumerator()
            {
                try
                {
                    foreach (var i in Query)
                        yield return i;
                }
                finally
                {
                    Dispose();
                }
            }

Since Query depends on Context, once Context is disposed we need to reset Query so a new one can be built (if possible). Note that we use an iterator here to return an IEnumerator<TSource>, rather than the usual IEnumerable<>.

Finally, we'll close out by explicitly implementing IQueryable:

            IEnumerator<TSource> IEnumerable<TSource>.GetEnumerator()
            {
                return GetEnumerator();
            }

            IEnumerator IEnumerable.GetEnumerator()
            {
                return GetEnumerator();
            }

            Type IQueryable.ElementType
            {
                get { return Query.ElementType; }
            }

            Expression IQueryable.Expression
            {
                get { return Query.Expression; }
            }

            IQueryProvider IQueryable.Provider
            {
                get { return Query.Provider; }
            }
        }

If this seemed relatively simple, you're right. We're just building a lazy-loaded Query proxy, with a bit of plumbing to clean up our Context.

Implementing ILazyOrderedQueryable

Not very exciting, but for completeness:

        public interface ILazyOrderedQueryable<TContext, TSource, TQuery>
            : ILazyQueryable<TContext, TSource, TQuery>, IOrderedQueryable<TSource>
            where TQuery : IOrderedQueryable<TSource>
        { }

        class LazyOrderedQueryableImpl<TContext, TSource, TQuery>
                : LazyQueryableImpl<TContext, TSource, TQuery>, ILazyOrderedQueryable<TContext, TSource, TQuery>
                where TQuery : IOrderedQueryable<TSource>
        {
            internal LazyOrderedQueryableImpl(ILazyContext<TContext> lazyContext, Func<TContext, TQuery> queryBuilder)
                : base(lazyContext, queryBuilder)
            {
            }
        }

LazyQueryable Factory

Consumers of this API should never need to know about these implementation details, so we can hide them behind a factory class:

        public static class LazyQueryable
        {
            public static ILazyQueryable<TContext, TResult, TQuery> CreateQuery<TContext, TResult, TQuery>(
                ILazyContext<TContext> context, Func<TContext, TQuery> queryBuilder)
                where TQuery : IQueryable<TResult>
            {
                return new LazyQueryableImpl<TContext, TResult, TQuery>(context, queryBuilder);
            }
            public static ILazyOrderedQueryable<TContext, TResult, TQuery> CreateOrderedQuery<TContext, TResult, TQuery>(
                ILazyContext<TContext> context, Func<TContext, TQuery> queryBuilder)
                where TQuery : IOrderedQueryable<TResult>
            {
                return new LazyOrderedQueryableImpl<TContext, TResult, TQuery>(context, queryBuilder);
            }
        }

Implementing ILazyContext

Again, we'll start with the interface:

        public interface ILazyContext<TContext> : IDisposable
        {
            TContext Context { get; }

            ILazyQueryable<TContext, TResult, TQuery>
                CreateQuery<TResult, TQuery>(Func<TContext, TQuery> queryBuilder)
                where TQuery : IQueryable<TResult>;

            ILazyOrderedQueryable<TContext, TResult, TQuery>
                CreateOrderedQuery<TResult, TQuery>(Func<TContext, TQuery> queryBuilder)
                where TQuery : IOrderedQueryable<TResult>;

            TResult Execute<TResult>(Func<TContext, TResult> action);
        }

Now we can start fulfilling our requirements:

1. Lazily expose the Context.

        class LazyContextImpl<TContext> : ILazyContext<TContext>, IDisposable
        {
            public Func<TContext> ContextBuilder { get; private set; }

            private TContext context;
            public TContext Context
            {
                get
                {
                    if (context == null)
                    {
                        context = ContextBuilder();
                        if (context == null)
                            throw new InvalidOperationException("Context built as null.");
                    }
                    return context;
                }
            }

2. Produce lazy wrappers to represent queries retrieved from a context by a delegate.

            public ILazyQueryable<TContext, TResult, TQuery> CreateQuery<TResult, TQuery>(
                Func<TContext, TQuery> queryBuilder)
                where TQuery : IQueryable<TResult>
            {
                return LazyQueryable.CreateQuery<TContext, TResult, TQuery>(this, queryBuilder);
            }

            public ILazyOrderedQueryable<TContext, TResult, TQuery> CreateOrderedQuery<TResult, TQuery>(
                Func<TContext, TQuery> queryBuilder)
                where TQuery : IOrderedQueryable<TResult>
            {
                return LazyQueryable.CreateOrderedQuery<TContext, TResult, TQuery>(this, queryBuilder);
            }

3. Execute an action on the context.

There are two ways to "complete" a query, and we need to clean up context after each. The first was after enumeration, implemented above. The second is on execute, implemented here:

            public TResult Execute<TResult>(Func<TContext, TResult> expression)
            {
                try
                {
                    return expression(Context);
                }
                finally
                {
                    Dispose();
                }
            }

4. Ensure the context is disposed as necessary.

We don't require that TContext is IDisposable, but we need to handle if it is. We also clear context to support reuse.

            public void Dispose()
            {
                var disposable = context as IDisposable;
                if (disposable != null)
                    disposable.Dispose();

                context = default(TContext);
            }

Constructors

With our requirements met, we just need a way to create our context. We provide two options:

            internal LazyContextImpl(TContext context) : this(() => context) { }

            internal LazyContextImpl(Func<TContext> contextBuilder)
            {
                if (contextBuilder == null) throw new ArgumentNullException("contextBuilder");

                ContextBuilder = contextBuilder;
            }

The former wraps an existing TContext instance in a closure, meaning every time ContextBuilder is called it returns the same instance. The latter accepts any delegate that returns a TContext. The most common such delegate would be a simple instantiation: () => new MyDataContext().

It should be clear now why we would want to clear our context on dispose. If ContextBuilder returns a new context instance each time, it's perfectly safe to discard of the old (disposed) context to trigger the creation of a new one. Conversely, if the builder returns a single instance, using the context after disposal would trigger an ObjectDisposedException or something similar.

LazyContext Factory

For consistency, we should also provide factory methods to hide this specific implementation:

            public static class LazyContext
            {
                public static ILazyContext<T> Create<T>(T context)
                {
                    return new LazyContextImpl<T>(context);
                }
                public static ILazyContext<T> Create<T>(Func<T> contextBuilder)
                {
                    return new LazyContextImpl<T>(contextBuilder);
                }
            }

Lazy Extensions

And last, but certainly not least, we're ready to reimplement our Use() extension methods:

    public static class Lazy
    {
        public static ILazyContext<TContext> Use<TContext>(this TContext @this)
        {
            return LazyContext.Create<TContext>(@this);
        }

        public static ILazyContext<TContext> Use<TContext>(this Func<TContext> @this)
        {
            return LazyContext.Create<TContext>(@this);
        }
    }

With several usage possibilities:

    var r1 = from x in new MyDataContext().Use() ...;

    Func<MyDataContext> f1 = () => new MyDataContext();
    var r2 = from x in f1.Use() ...;

    var r3 = from x in new Func<MyDataContext>(() => new MyDataContext()).Use() ...;

    var r4 = from x in Lazy.Use(() => new MyDataContext()) ...;

Or maybe we can make it even easier. Maybe...

Solutionizing .NET (Keith Dahlby)

posh-git: A PowerShell Environment for Git

SPWeb.AssociatedGroups.Contains Lies

Extension Methods on Types You Own?

Quick Tip: Parse String to Nullable Value

Selecting Static Results with Dynamic LINQ

DynamicExpression.CreateClass

Dynamic to Static

Selecting Anonymous Types

HTTP Error Codes in WatiN 1.3

Private Strikes Again

Script to Enable HTTP Compression (Gzip/Deflate) in IIS 6

Is Functional Abstraction Too Clever?

Refactoring with Iterators: Prime Factors

Hacking LINQ Expressions: Join With Comparer

The Problem

The Goal

The Hack

Hacking LINQ Expressions: Select With Index

The Problem

The Goal

The Hack

Functional Construction for ASP.NET Web Forms

Expressions, Expressions, Expressions

Simplifying LazyLinq

A Note on IOrderedEnumerable<>

Introducing LazyLinq: Queryability

Learning from Queryable

Querying ILazyQueryable

Introducing LazyLinq: Internals

Implementing ILazyQueryable

Implementing ILazyOrderedQueryable

LazyQueryable Factory

Implementing ILazyContext

1. Lazily expose the Context.

2. Produce lazy wrappers to represent queries retrieved from a context by a delegate.

3. Execute an action on the context.

4. Ensure the context is disposed as necessary.

Constructors

LazyContext Factory

Lazy Extensions

A Note on `IOrderedEnumerable<>`