It seems that they do if they reference members of the declaring type. This makes perfect sense now that I think about it, but I didn't think about it earlier and wrote some code that caused a memory leak. I had an object acting as a singleton and referencing a delegate instance that was created from an object acting as a non singleton. Bam! Memory Leak.

I setup a little test to demonstrate. Here is a class that has two methods which return delegate instances:

public class TestSource
{
    private string internalValue = "test";

    public Func<bool> GetFunc1()
    {
        return () => 1 == 1;
    }

    public Func<string> GetFunc2()
    {
        return () => internalValue;
    }
}

Notice that GetFunc1() doesn't have any references to internal members, but GetFunc2() does. Here's the test class:

class Program
{
    static void Main(string[] args)
    {
        // hold no references to TestSource
        TestFuncInstance(new TestSource().GetFunc1());
        TestFuncInstance(new TestSource().GetFunc2());

        Console.WriteLine("Press Enter to continue.");
        Console.ReadLine();
    }

    private static void TestFuncInstance(Delegate func)
    {
        Thread.Sleep(1000);// give some time for GC

        Console.WriteLine(string.Format("Method: {0}",func.Method));
        Console.WriteLine(string.Format("DeclaringType: {0}", func.Method.DeclaringType));
        Console.WriteLine(string.Format("Target: {0}", func.Target ?? "null"));

        Console.WriteLine();
    }
}

This creates two separate instances of TestSource and passes the result of the two GetFunc methods to a test method. Notice that there are no declared variable references to the TestSource object. Here's the output of the test:

Method: Boolean <GetFunc1>b__0()
DeclaringType: TestingDelegates.TestSource
Target: null 

Method: System.String <GetFunc2>b__2()
DeclaringType: TestingDelegates.TestSource
Target: TestingDelegates.TestSource 

Press Enter to continue.

I know this test isn't very scientific, but you'll see that Func1's target is null while Func2's target is not. Func2 has to hold a reference to the declaring object so that it can do it's job when invoked. Func1 does not need a reference, and seems to free up the declaring object to be garbage collected. This is definitely something to keep in mind when passing around delegates.