Polyglots R Us

Newbie Win32 Development Using Python

Ryan Svihla — Sat, 16 Jan 2010 04:35:33 GMT

First a disclaimer. I’m not in anyway shape or form an experienced win32 systems programmer. I’ve always done application development or systems administration scripting. That in-between area where you get out your C/C++ compiler and start dealing with pointers is completely alien to me. This has created problems in the past when I need to do something outside of the scope of C# and found myself staring at MSDN docs in C trying to extrapolate the equivalent C# code, and typically being disgusted at having to use hand rolled Structs with calls to Marshall.GetLastWin32Error().

Worse still I end up with a nasty implementation that requires slow, fragile integration tests to verify behavior or very verbose mirror interfaces where I’m testing order of calls. Look at the following code sample to read a reparse point’s target directory from code project:

 // Allocate a buffer for the reparse point data:

Int32 outBufferSize = Marshal.SizeOf(typeof(REPARSE_GUID_DATA_BUFFER));

IntPtr outBuffer = Marshal.AllocHGlobal(outBufferSize);

try

{

 // Read the reparse point data:

Int32 bytesReturned;

// WOW what a signature, it's requiring blank data! 

// Looking at MSDN the signaure contains input and output buffer and size

// for both, meaning I'm always sending something as IntPtr.Zero

Int32 readOK = DeviceIoControl( hFile,FSCTL_GET_REPARSE_POINT, IntPtr.Zero, 0,

                                outBuffer, outBufferSize, out bytesReturned,IntPtr.Zero);

if (readOK != 0)

   {

   // Get the target directory from the reparse 

   // point data:

   REPARSE_GUID_DATA_BUFFER rgdBuffer =

   (REPARSE_GUID_DATA_BUFFER)Marshal.PtrToStructure(outBuffer, typeof(REPARSE_GUID_DATA_BUFFER));

   targetDir = Encoding.Unicode.GetString( rgdBuffer.PathBuffer,rgdBuffer.SubstituteNameOffset,

                                          rgdBuffer.SubstituteNameLength);

	//removed a bunch more stuff to format the result

// Free the buffer for the reparse point data:

 Marshal.FreeHGlobal(outBuffer);

In the past when building these kind of things for less complicated code I just would mirror out all the untestable code like Marshal , DeviceIoControl and CloseHandle calls with interfaces, then create a production wrapper. This however leads to very long tests, with lots of specifics about the internals of the interaction, in the above example the support code looks like:

    public interface IMarshal

    {

        void FreeHGlobal(IntPtr buffer);

        Object PtrToStructure(IntPtr pointer, Type structure);

        Int32 SizeOf(Type structure);

        IntPtr AllocHGlobal(int cb);

    }

    public class ProdMarshal : IMarshal

    {

        public void FreeHGlobal(IntPtr buffer){

            Marshal.FreeHGlobal(buffer);

        }

        // you get the idea from here 

    }

Not terribly pretty mocking and testing this leads to is it? This isn’t also counting the win32 calls that are needed to setup and finalize the data around this. Before its all said and done to make this “testable” you end up with 2 interfaces, 2 wrapper production classes, and very heavy interaction specific mocking to even handle this basic idea of getting a value from a file system object. Before you try and point out “well you can reuse that logic later” look at the size of Marshal and ask yourself do you really want to mock all that out?

Adding insult to injury your mocks have to be very good to replicate the behavior of the actual real thing, and it varies greatly across platforms. Windows 7 and server 2008 r2 allow reparse points (or symlinks in the this case) from directory to directory, but Vista and 2008 r1 are a no go and will fail.

No wonder the systems developers that I know completely turn there nose up at unit testing and instead focus on slow integration or systems tests. The abstractions take time to make happen and the code itself is just not testable without lots of mirrored interfaces.

So not wanting to spend all my time just writing mirrored interfaces day in and out for win32 calls, nor liking the idea of turning my back on unit testing, I decided to take a look at dynamic languages win32 support.

First I looked at Ruby and the win32 libraries I saw didn’t give me the above functionality I needed with the minimal amount of hassle I wanted. DeviceIoControl up there for example with its stunningly bad signature is still pretty rough in Ruby as they have a one to one signature mapping.

Python however has had for a very long time strong win32api support and removes some of the awful pain that you have to deal with. Below is the equivalent of the code above, again with items stripped out for brevity before and after it, but they’re roughly computationally equivalent and the Python version wins out more the more I include.

output_buf=win32file.AllocateReadBuffer(winnt.MAXIMUM_REPARSE_DATA_BUFFER_SIZE) 

#notice how I don't have to pass in size of the buffers now. I still have the need to 

# pass None in for the input buffer here however

buf=win32file.DeviceIoControl(h, winioctlcon.FSCTL_GET_REPARSE_POINT,None,OutBuffer=output_buf, Overlapped=None) 

fixed_fmt='LHHHHHH' 

fixed_len=struct.calcsize(fixed_fmt) 

tag, datalen, reserved, target_offset, target_len, printname_offset, printname_len = \ 

     struct.unpack(fixed_fmt, buf[:fixed_len])

It’s still not pretty, but its briefer, and I can actually stub out or test struct.calcsize, AllocateReadBuffer and DeviceIoControl with no need to mess around with setting up lots of test harness code. Using my current mocking framework the test code looks something like this:

alloc = mock.replaceWithMock(win32file, "AllocateReadBuffer")

buffer = object()

alloc.stub(buffer)

#removed extra setup code for calcsize and DeviceIoControl

getTargetDir()

mock.assertCalled(win32file, "AllocateReadBuffer")

Much cleaner, much lower friction and so I’m satisfied with Python for win32 development so far. I’ll give C# another look when I have the dynamic keyword and I don’t have to make explicit mirror interfaces for everything that’s untestable.

Adventures in Meta programming in Python: im_class, import, getattribute

Ryan Svihla — Fri, 08 Jan 2010 05:03:00 GMT

With my recent work in Python win32 programming I’ve had a real need for AAA style mocking framework. Unable to find anything that I’ve been completely happy with I started my own simple mocking framework and got to learn some Python style meta programming in the process. I’ve found out a lot about the depth of the Python 2.x object model over the last 2 weeks and here are some of the nicer things I’ve found (updated as per Chris Taveres below):

function_class = MyClass.foo.im_class

This is useful if you pass in an instance method somewhere and you want to be able to access its attached class. Decorators on instance methods normally use this, but I needed it to get a nicer syntax on my asserts, I wanted this:

assertCalled(FakeRepo.get)

So now when I want to track what class is passed in I just have to start with the following:

def assertCalled(method):

    classdef = method.im_class #I get the method and the class in one parameter!

module = import(“foomodule”)

this creates a module import by string. This becomes super powerful say if you wanted to replace functionality on a module, but didn’t know its name before hand. In my case I wanted to be able to patch in a replacement method for mocking and then put it back when I was done, and I didn’t want to have to keep passing in the module name on resets. The only downside to it so far is it behaves a little quirky with packages and you have walk your module hierarchy to get it to do what you want

example of patching originals back into the module:

for mod, original in _originals.items():  #key is a combination of module name and methodname, value is the original function itself

    modulename = mod[0]

    methodname = mod[1]

    module = __import__(modulename)  #this is the magic. if module name has no package aka : “barmod” this will work. if its “foopackage.barmod” it’ll import the “foopackage” 

    for i in modulename.split(".")[1:]: #works down the chain 

        module = getattr(module, i) #resets the module variable with the lower hierarchy module 

    setattr(module, methodname, original) #actually pass the original function object back onto the module.

MyClass.getattribute = interception

__getattribute__ is called when any attribute is accessed on a class including __getattribute__ itself which is a bit silly. This was the primary engine of my mocking framework as it allowed me to record all calls to the methods on a mocked class. Just remember when you use this to have it not call itself or you’ll be in endless recursion!

example:

class MyClass(object):

    def stuff(self):

        pass

def interceptor(self, name):

    if name == "__getattribute__": #guard condition against calling itself

        return object.__getattribute__(self, name)

        #whatever interception logic you need here

MyClass.__getattribute__ = interceptor

m = MyClass()

m.stuff() #interception logic will be called here

I’m just starting to dip into the Python object model now and I’ll try and share what I find over the next few weeks.

Take 3: Python, ISP, IoC, and OCP need a fundamental rethink.

Ryan Svihla — Fri, 20 Nov 2009 19:23:00 GMT

In response to Julian’s thoughtful Dynamic Languages and SOLID Principles I’d have to argue he is about 95% there but is missing the last critical links needed to view this in a whole different light. For ISP, Julian says it all:

Ultimately, I don't think ISP is changed by Python, it's just kind of irrelevant, for better or worse.

Exactly! Why even bother mentioning in the context of dynamic languages. A developer that understands it fully and one that does not will code the same way. Whereas LSP is still taught in dynamic languages, I’ve never seen it called that but I’ve seen many admonishments about subclass consistency, and not making client code aware of the differences, and is something that requires mentioning. With ISP we could ignore it’s entire existence and developers would code the same.

For OCP:

Well, the open closed principle is a goal, not a design practice, but let's take a look at the danger points:

    * You can't have non-virtual methods, so Python wins this hands down.
    * Your variable can't be made too specific, so you're safe there.
    * You can still compare against hard-coded values. It's just as easy to get this wrong in Python as it is in C#.
    * Same holds true for Law of Demeter violations. If you pass the wrong object around, your code will be just as fragile in Python as in C#.

Python certainly reduces the scope for some OC violations, but you've still got lots of rope to hang yourself. Think you still need to bear the goal in mind.

Bolding emphasis is mine. Enough of OCP is stripped and the primary ways its taught and demonstrated do not apply to the language at all. Focus on Law of Demeter, but having something that stands for “Open for extension” whenever everything is open and “Closed for modification” when nothing is closed would fall on completely deaf ears when presenting any of the associated ideas to a Python developer.

For Dependency Inversion/ IoC:

Let's quote Ryan:

Now all calls in this runtime, from any module, that reference the Output class will use XmlOutput or HtmlOutput instead.

Yes, but what if I wanted only half of them? Maybe there's Python techniques I don't know about (I'm barely competent in the language) but as I see it, I'm going to need to change the code. I don't think that dependencies can "always" be injected. They can only be done when it won't cause damage. In his case, he's worrying about testability. That's fine, but we all agree there's more to DI than testability.

Ok firstly, I didn’t just mention testability, in fact that the code you’re referencing there was no unit test for, it was about flexibility. Secondly, call Output an “Abstract Class”,and then call XmlOutput and HtmlOutput concrete types and then add them to an IoC container like Windsor then let me ask yourself the question you asked me “what if i wanted only half of them”. There would be a need for custom code to make this resolution work for this custom circumstance. All I’m advocating is taking that same exact custom code and place it somewhere else in Python (yes you can override per method, per instance, etc, etc).

I think there needs to be a principle that addresses the benefits of explicit interfaces and well formed custom composition cases…and in those cases DI makes a lot more sense, but certainly not as a defacto convention and certainly is not required to provide flexibility to existing implementations. It certainly becomes apparent when you start looking into the actual way IoC works in static languages its all too similar to how a dynamic language runtime works. That code you’d use in a static language IoC to apply AOP, or custom composition rules, could just as easily be applied in a component in Python, because the primary mechanism for an IoC to work is as a factory, and the benefits and features they bring are all derived from having access to how objects are created and then returned, when you always have access to how objects are created and returned anyway as in dynamic languages, why should you implement it a second time. 100% DI’d code will be more effort to maintain because it would be in C# too…if not for that fact the language is so “structured” that the flexibility and testability benefits gained far outweigh the slightly increased cost (which is only brought down with the awesomeness of auto registering components). In a language already with flexibility and testability whether you use DI or not AND without auto registering IoC containers the cost to maintain a full DI code base is I’d argue unacceptable.

Summary, for SOLID to make any sense whatsoever in a dynamic context it has to amended and modified to fit that world. Maybe make SOLID into SELL for:

S.RP

E.xplicit empty class for required overrides

L.SP

L.aw of Demeter

Maybe there is something containing other more valid concepts for the dynamic world that I’m probably not qualified to design or come up with. Anyway, I’ve enjoyed the conversation and I hope it brings about a firmer understanding of SOLID, and whatever we’ve not come up with yet for dynamic languages.

Take 2: Why we use SOLID in static languages and how we get the same functionality for cheap in dynamic languages

Ryan Svihla — Thu, 19 Nov 2009 22:46:17 GMT

One of the things we do pretty well at Los Techies is explaining SOLID principles and why they make our code more maintainable and if you’re not familiar with our work on SOLID, read Chad Meyer's post to get an understanding on the Los Techies perspective. Now that you’ve taken a bit of time to read that the following chart takes SOLID’s benefits and then shows how they apply to dynamic languages:

Principle	Primary Benefit	Secondary Benefit	What dynamic langs do to replace said principle
Single Responsibility Principle (SRP)	Classes are in understandable units	Testability	Nothing
Open-Closed Principle (OCP)	Flexibility	Guidance	can override anything globally (monkey patch)
Liskov Substitution Principle (LSP)	Not bleeding subclass specific knowledge everywhere	Least surprise	Nothing
Interface Segregation Principle (ISP)	Interfaces are in understandable units	Less work for interface implementation	everything’s an interface (duck typing)
Dependency Inversion Principle(DIP)	Flexibility	Testability	can override anything globally (monkey patch)

The above is my opinion and I’m sure people will beat me to death because I’m just some dude not many people know about, but I think I’ve got a grasp of what SOLID means in C# and I’ve grappled with it from a practical perspective.

You’ll note when it comes to LSP and SRP dynamic languages really give you nothing at all, but OCP, ISP, and DI benefits wise are supplanted by language features in python, ruby, etc. How does this happen? In the case of ISP and “duck typing” this should be pretty obvious, but in the cases of OCP, DIP

DIP/OCP testability achieved through C#:

public class FooBar{

    private IFooService _service;

    public FooBar(IFooService service){

          _service = service;

    }

    public void DoFoo(){

        _service.run();

    }

}

//testcode

[TestFixture]

public class FooBarSpec{

  [Test]

  public void should_call_fooservice(){

     var mockfoo = new Mock<IFooService>();  

     var foobar = new FooBar(mockfoo.object);

     foobar.DoFoo();

     mockfoo.Verify(foo=>foo.run())

   }

}

DIP/OCP equivalent testability achieved through Python:

class FooBar(object):

    def DoFoo(self):

        service = FooService()

        service.run()

#test code

import unittest

class TestFooBar(unittest.TestCase):

    def runmock(self):

        self.fooservice_run_was_called = True

    def test_do_fooservice_is_called(self):

        originalrun = FooService.run #storing the original method in a variable 

        FooService.run = self.runmock  # I'm globally overriding the method call FooService.run AKA Monkey Patch

        FooBar().DoFoo()

        FooService.run = originalrun #globally put it back to where it was AKA Monkey Patch

        assert self.fooservice_run_was_called

Now rightfully you may say the Python version has a few more lines of code, and it certainly can be considered ugly…but don’t you see that I replaced an entire mocking library in a few lines of code. There are also mocking libraries and monkey patch libraries for Python that would shorten my line of code count further. Testability and flexibility dovetail, but the global accessibility of class definitions is understandably hard to grasp for the static mindset (it took me months), so we’ll continue on to flexibility:

public interface IOutput{

  void save(string fileloc, ResultObjects[] results);

}

public class XmlOutput:IOutput {

   public void save(string fileloc, ResultObjects[] results){

    // place xml specific code here

}

public class HtmlOutput: IOutput {

   public void save(string fileloc, ResultObjects[] results){

    // place html specific code here

}

public class TestRunner {

  private IOutput _output;

  private ITests _tests;

  public StoreTestResults(IOutput output, ITests tests){

    _output = output;

    _tests = tests;	

  }

  public void RunTestsFromDir(string testdirectory, string outfile){

    var results = _tests.GetResults(testdirectory);

    _output.save(outputfile, results)

  }

//later on we use IoC containers to make this chain easy 

pubilc static void main(string[] args){

  // imagine contextual resolution is happening intelligently 

 // on the resolver because someone did some cool stuff in windsor

  var runner = testcontainer.Resolve<TestRunner>();

  runner.RunTestsFromDir(args[0], args[1]);

}

In Python with “monkey patching” everything is “Open For Extension” and dependencies can “always” be injected. A code sample based on my last blog post but extended to include some more complete client code follows:

#in module outputlib

class Output(object):

  def save(fileloc, results):

    pass

class XmlOutput(object):

  def save(fileloc,results):

     """xml specific logic here"""

     pass

class HtmlOutput(object):

  def save(fileloc,results):

     """xml specific logic here"""

     pass

class TestRunner(object):

  def run_tests_from_dir(testdirectory, outfile):

    tests = Tests()

    output = Output()

    output.save(  outfile  , tests.get_results(testdirectory)

#rest of the script

import sys

import outputlib as o

testdirectory = sys.argv[1]

outfile = sys.argv[2]

if outfile.endswith("xml"):

  o.Output = o.XmlOutput #globally override calls to Output with XmlOutput. AKA Monkey Patch

elif outfile.endswith("html"):

  o.Output = o.HtmlOutput #globally override calls to Output with HtmlOutput AKA Monkey Patch

runner = o.TestRunner()

runner.run_tests_from_dir(testdirectory, outfile)

Again this is longer code than the c# example.. but we’ve replaced an entire IoC container and whatever custom code we would have had to add to create custom dependency resolver for selecting HTML or XML output options has now been replaced by a simple if else if conditional at the beginning of the code base execution. Now all calls in this runtime, from any module, that reference the Output class will use XmlOutput or HtmlOutput instead, all with “monkey patching”.

Hopefully, from here the rest of the dominos will fall into place once you get the concepts above, and the things we rely on SOLID for are in some cases already there in dynamic languages. Consequentially, my dynamic code looks (now) nothing like my static code because I no longer have to use DI/IoC to achieve my desired flexibility and access to use proper unit tests.

I recant my IoC! IoC containers in dynamic languages are silly.

Ryan Svihla — Mon, 16 Nov 2009 22:28:33 GMT

After a year or so of solid Alt Dot Net infection (as far as infections go its a pretty awesome one to have), I decided to give Python a try again for more than one off sysadmin tasks, and to actually dive into it as a newly minted “Agilista”.

However, I had a problem..there were no non-painful IoC containers in Python (sorry to the other authors of IoC frameworks in Python like Spring Python and Snake Guice, I know you try and I respect the effort). Ultimately, I could not imagine coding anymore without something to handle all my registration for me, that’d dynamically inject in my dependencies, give me hooks for contextual resolution, and give me rich interception support.

So I built my own, it was painful still, but I had some ideas to move it in an Convention Over Configuration direction, and ultimately get within shooting range of what fancy stuff we’ve come to expect in .Net that Windsor, StructureMap and many others provide.

Now as I got into the fancier aspects of dynamic languages, open classes and the ability to override all behavior easily I get it…a dynamic languages runtime is like a bit IoC container.

Now I’ve heard other people say this many times but rarely with explanation or example, or focus on frankly silly things like “Dependency Injection adds too many lines of code” which is a bit melodramatic. Two arguments to a constructor, two fields ,and then avoiding having to call new is not “adding too many lines of code”, especially with templates, IDE’s, scripting, etc to cut down the actual typing load.Today I’m going to actually try to explain how languages like Python, Ruby etc give us the same awesomeness we’ve come to expect in things like Windsor..but at a much cheaper cost of learning and dev time.

Take a typical resolution scenario where you want to output to a different file format depending on command line switches. With an IoC container you can either:

Change the resolution argument to load a different concrete type:

if(arg == "XML")
{
  container.Register(Component.For<IOutput>().ImpmentedBy<XmlOutput>());
}
else if(arg == "HTML")
{
  container.Register(Component.For<IOutput>().ImpmentedBy<HtmlOutput>());
}
else
{
  container.Register(Component.For<IOutput>().ImpmentedBy<NullOutput>());
}

or resolve different arguments or keys using a service locator approach in later client code (thereby depending on the container)

public void output(IKernel container, string key)
{
var output =container.Resolve<IOutput>(key);
output.save();

}

or implement a custom implementation of the resolver system (which I’ll leave out for the sake of brevity, but it’s not instant code). Also to do all this you have to depend on interfaces, add all your interchangeable code to the constructor and life is grand. You do this for many reasons in static languages, its the only way to get easy testability and code that is open to extension. In dynamic languages its always open for extension and easy to test . Let me demonstrate:

import outputlib as o

def outputselect(arg):
if arg == "XML":
o.Output = XmlOutput
elif arg == "HTML":
o.Output = HtmlOutput
else:
o.Output = NullOutput

def saveoutput():
o.Output().save() #will save whichever

Contextual resolution in a nutshell, and throughout your code if need be. “Interception” is even easier, take a look at http://docs.python.org/dev/library/functools.html and then start playing you’ll see you can trivially apply logging and security to methods without explicitly adding it. A short logging example follows:

import types
import functools

#applies a cepter to each non-underscored method.
def wrapcls(cls, cepter):
    publics  = [ name for name in dir(cls) if not name.startswith("_")]
    methods = [getattr(cls,method) for method in publics if type(getattr(cls,method)) == types.MethodType ]
    for method in methods:
        intercepted_method = cepter(method)
        setattr(cls, method.__name__, intercepted_method) #attaches intercepted_method to the original class, replacing non-intercepted one

#the magic all happens in the functools.wraps decorator
def loggingcepter(func):
    @functools.wraps(func)
    def logafter(*args, **kwargs): #for csharp devs view this as an inline delegate
        result = func(*args, **kwargs) #invoking function
        print "function name: " + func.__name__
        print "arguments were: "
        for a in args:
            print repr(a)
        print "keyword args were: "
        for kword in kwargs:
           print repr(kword) + " : " + repr(kwargs[kword])
        print "return value was: " + repr(result)
        return result
    return logafter

#default boring repository class
class StorageEngine(object):
    @property
    def connections(self):
        pass

    def _sessioncall(self):
        pass

    def create(self,user):
        print "running create now from the method"

    def delete(self, id):
        print "running delete now from the method"
        return "deleted from the database"

    def get(self, id):
        pass

    def __init__(self):
        pass

#placeholder storage object
class User(object):
    pass

wrapcls(StorageEngine ,loggingcepter)
repo = StorageEngine()
print "calling count this should not be intercepted"
cnncount = repo.connections
print "now get should be intercepted"
repo.get(1)
print "we should see keyword arguments here"
repo.delete(id=2)
print "session call should not be intercepted"
repo._sessioncall()
print "create should be intercepted and we should see a User object"
repo.create(User())

Running the above script should result in the following output

So this is all very cool, but what does it mean or why do I care? For those of us used to using a proper IoC container like Windsor or StructureMap, we’ve gotten used to have capabilities like the above easily available to us when we’ve needed them. It’s nice to find that in Python (or really any dynamic language ) we can easily build our own similar functionality that we’ve come to depend on. We’re never coupled and we’re always able to test and mock out behavior. It was a long time coming but I think I finally get it now.

Alt.Net Evangelism And What We Could Do Better

Ryan Svihla — Mon, 14 Sep 2009 00:06:00 GMT

If we assume our goal as the Alt.Net community is to improve software development as a whole, we're doing a pretty bad job. Not that many people have even heard of the concepts we espouse, and those that have but didn't convert become hostile to our ideas.

My theory is because we focus so much on things that at the surface seem extraordinarily complex. Our terminology while correct, is acryonym laden and hard to grasp the real meaning. We ourselves sometimes lose site of why we even care about some of those principles, why we enacted them in the first place and what pain they caused to begin with.

I'm not the first person to say or think this, but this has really hit home with me lately since now I get to talk to developers who are brilliant, talented and overall effective at what they do but who consider XP+SOLID a bunch of "enterprise junk" that fights against "natural coding" and my personal favorite "Unit testing is just testing your tests" .

I've tried to talk to people in a reasonable way when I hear things like this, but usually before I know it, the alphabet soup of acronyms is out of my mouth, BDD, DDD, on and on. Even as a hungry student those terms shut my brain down when I first heard them, so why would I expect someone who feels they know what goes into software development to have a different reaction?

Since my goal is to do my small humble part to improve software development as a whole I'm going to try something new. If someone feels pain in the development process, then I'll speak up with a solution and focus my language on the solution only. Because really thats what all these ideas and concepts are, solutions to big pain points.

The terms we use to describe them are unfortunately for the already intiated. Inversion Of Control, did any of you understand that without looking at code examples first? Behavior Driven Development, how many of you thought you knew what that was, practiced it, then realized that it means something else. Just spouting off a bunch of acryonyms only excites the already curious who just have to know what they mean and for better or for worse they do not make up the bulk of the software development community.

As for blogging thats a tough break, I use blogging as part envangelism, part bouncing ideas off my peers and superiors. For the later I do need to actually use our alphabet soup to communicate, but for evangelism I'm afraid I contribute to turning off devs who are only googling for a solution.

Feedback?Ideas?

Playing With Haskell

Ryan Svihla — Wed, 02 Sep 2009 03:53:00 GMT

My tech book club is starting to look at Haskell. I'm as big of a language addict as anyone so I wanted to dive in early and this is the overview of what I've found so far

Statically Typed But Dynamically Inferred

for those of you familiar with Boo this is a familiar concept to you. Once a type is made that type will not autoconvert. For Example:

let i = 6 --this is now going to be an integer

let f = 1.2 --this is now a float

i + f /--will throw exception

Cannot set variable twice

this is from what I understand something that all functional languages share some concept of. (Erlang views the second attempt to set a variable as a "match" request).

x = 10

x = 11

when running the source file results in:

Multiple declarations of `Main.x'

Declared at: functions.hs:1:0 functions.hs:2:0

Failed, modules loaded: none.

My Least Favorite Thing In Python AKA Significant Whitespace

In the above code file in my previous example I had an error with an extra whitespace hiding about. I got used to this in Python by using soft tabs and being aware of phantom white space. Not happy about this.

No Loops!

The two primary ways I've found to replace loops are recursion and map. Map is a standard of functional languages and pretty easy to grasp. It's just a way of saying "run a specified function on each item in the list". A quick example of map is as follows:

doubleList ls = map doubleInt ls -- function named "doublelist" with parameter of ls
where doubleInt x = x*2 --doubles x which represents each value in list

doubled = doubleList [1,2,3,4,5] --set global variable with function applied

At A Glance

Haskell seems to have an easier to learn syntax than Erlang, while still being less ceremonial than other languages I've used. I kinda dig the no parenthesis syntax of functions, and of course I don't miss foreach loops with map. Finally, I'll be anxious to find out what the multi-core story is vs Erlang.

Simple BDD/TDD

Ryan Svihla — Wed, 02 Sep 2009 02:47:00 GMT

Todays theory is most tests and specs should be very short (2-3 lines), have at most a setup for context establishment, avoid the majority of test framework features as they should be used as an exception and not as a rule. Note: I practice BDD nowadays and do not like using the term "test", but these rules apply for BDD as well as TDD, and I imagine most of you practice TDD so I'll be using primarily TDD terminology.

If you look at my first attempts at TDD they are very similar to my current BDD work with only changes in language. But the contexts, conditions and tests are similar. Most importantly they're easy to read. They're all very short, very descriptive, make limited use of arguments in the NUnit Attributes, and make very limited use of mocking.

Of course along the way I went down all sorts of terrible wrong paths, and with the help of Gerrard Meszaros plus some of my own experimentation I'm much happier with the time it takes to implement tests and the limited pain in refactoring my code.

Bad ideas I had along the way include:

Ran everything in TestFixtureSetup and TestFixtureTearDown or equivalent. Tests were all linked to one another
Mocked EVERYTHING and doing so with Record/Replay was double awful. Simple refactorings would break way more tests than they should. Test methods were 8-40 lines and unreadable at that.
Tried to sort everything by category with NUnit attributes. Took a fair amount of time for no payoff I can see (think about changing a category name when all you have is seemingly unrelated strings)
Tried to create "Story Classes" for testing. I know some people like this approach, I find them to be heavily coupled, fragile and noisy. Also this made finding individual examples of usage of individual classes difficult as now its "rules" were spread everywhere

Summary point avoid some of my mistakes, if you write short unit tests now..thats probably a good thing. If you are struggling with testability of your classes and you find your tests getting big, try and stop, sit down and make sure you are testing "one thing" and not really a big thing with many small pieces.

Dynamic Loading of .Net Assemblies

Ryan Svihla — Mon, 31 Aug 2009 22:29:00 GMT

Problem: You have dependencies which can be in a number of directories outside of the directory your app is running in. DotNet AppDomains are heavy to say the least and very high in ceremony. You do not want to deal with secondary app domain creation.

Solution: Override AssemblyResolve event on your current app domain.

    public class AppDomainLoader : IAppDomainLoader

    {

        private ResolveEventHandler _searchAssemblyMethod = _searchDir;

        public ResolveEventHandler FindAssemblyMethod

        {

            set { _searchAssemblyMethod = value; }

        }

        private static string[] _directoriesToSearch;

        private  static Assembly _searchDir(Object o, ResolveEventArgs e)

        {

            foreach (var s in _directoriesToSearch)

            {

                var filetoload = s +  e.Name.Split(',')[0] + ".dll"; //bad one liner that parses the first part of the assembly name and adds .dll at the end

                if (File.Exists(filetoload)) //sees if the file is even in the directory

                {

                   return Assembly.LoadFrom(filetoload);  //returns assembly once found, a bit naive and does not handle version conflicts well.

                }

            }

            return null;

        }

        public void Search(string[] directoriesToSearch)

        {

            _directoriesToSearch = directoriesToSearch; //sets directories to search to allow entry point for dynamic searching

            AppDomain.CurrentDomain.AssemblyResolve -= _searchAssemblyMethod; //cleans up previous events, may not be needed.

            AppDomain.CurrentDomain.AssemblyResolve += _searchAssemblyMethod; //registered the event

        }

    }

Now whenever there is a missing Assembly, the _searchAssemblyMethod will be fired, and all directories given to the Search method will attempt to find an assembly with the matching name.

Focused and Narrow; Superficial and Broad

Ryan Svihla — Sun, 30 Aug 2009 15:12:00 GMT

In my 20's I got exposed to a large variety of work situations and was able to succeed in a variety of technical fields (admin, developer, hardware tech, etc). I had the time back then to not have to choose which direction I went with things, I went broad but deep into each field I was interested in. Every couple of years I'd go head long into another field and read dozen books, setup labs experiment like crazy, use whatever connections I'd made to get my foot in the door somewhere and then make sure I delivered in big ways, and most importantly keep reviewing what I'd done in the past to make sure it actually did work and did work well.

This obsessive learning habit had a great side effect, as long as I stuck to slightly related fields I learned exponentially more. Learning about networking made helping customers with a variety of issues far easier when I was just a lowly computer tech. When I was trying to automate customer installations in a repeatable way, learning software development even at a trivial level made things possible that were not otherwise. I got to see how everythign interconnected.

Nowadays though I have a busy life, I no longer can work 80+ hours a week year round, while studying, practicing or experimenting with the rest of my time. I now have to choose which way to do I go. I think I'm mostly going "Focused and Narrow" on a few subfields that are closely related to one another. I will focus for the near future on systems development and virtualization, I'll still probably be a language addict and wanting to do it a number of different ways, but I think for awhile I need to consildate my limited time and energy. I hope I don't lose my larger view and perspective on things, but I doubt I can any longer keep up with new versions of Postfix, Exchange, or Qmail, motherboard slot types, RAM, or whatever things that I used to know in much greater detail than I do now.

“Hero Mode Refactoring” AKA Enemy Of Brown Field Development

Ryan Svihla — Fri, 14 Aug 2009 06:48:56 GMT

What do I mean by “Hero Mode Refactoring”? We’ve all worked with code that wasn’t easily made testable. Most of the time we see a big ball of mud and have no idea where to begin. Sometimes however we have an epiphany, a _great_idea_ , a “cold fusion” moment (that ends up being poignant in so many ways).

Filled with the excitement of our vision we dive head long into our work, replacing concrete classes with interfaces, removing big swaths of redundant code, and in general making changes you should only make with a an already unit tested code base. Before we know it, 5 things are broken in weird ways, and it now takes us days of manual verification to figure out what works and what doesn’t. That is hero mode refactoring at its best.

This situation reminds me a great deal of the difficulties I had with getting back in shape. I was a pretty decent athlete in my college days, but the real world had other ideas. Years later I tried to get back in shape several times, each time I tried my hardest right off the bat, and most of the time got injured in the process. It wasn’t until age had given me some patience that I was able to actually get back in the gym in a consistent way. I had to work on the areas I could handle in a gradual way, instead of going 100% day 1. I wasted years trying to get to 19 year old me in a month, but it only took me 6 months going the gradual way to be in decent shape.

So too our codebases will behave if we try and go full out on day 1. That code didn’t get that way overnight, and usually cannot be fixed overnight. Take a “siege” mentality into your code maintenance and realize this is a battle best fought gradually.

Introducing SpecMaker “Rspec style” BDD in C#

Ryan Svihla — Sat, 08 Aug 2009 12:00:00 GMT

So I’m certain this will be met with mixed response, because really .Net already has several decent BDD frameworks and many of you will chastise me for adding yet another framework when really BDD has nothing to do with what testing framework you use. So why you ask?

Most of the BDD frameworks I’ve looked at are Acceptance style and trying to make stories into executable code (NBehave, StoryTeller, Fitnesse.Net ,Acceptance, etc). I want something that describes to other developers in a behavior centric way what my code is doing (like RSpec’s default DSL does). This is not aimed at business analysts.
The other RSpec style framework for C# I’ve played with while very nice, did not go over well with people I’ve tried to introduce to BDD.
Using NUnit in a context BDD style is normally what I do but produces a lot of artifacts, is underscore heaven and provides no guidance to newer practitioners of BDD.

With all this in mind how does SpecMaker improve our situation at all? First lets look at how I might approach a BDD test with NUnit

   [TestFixture]

    public class SpecGameWhenStartingUp

    {

        private Game _game;

        private void startGame(){

           // left out for clarity

        }

        [SetUp]

        public override void SetUp()

        {

           startGame();            

        }

        [Test]

        public void should_have_3_lives()

        {

           Assert.AreEqual(3, _game.lives);

        }

     }

On the surface there isn’t much wrong with this. Asserts are less than ideal (Rspec matchers would be nice), underscores on my “should_” are so so, context in the class name leads to lots of classes and me playing around a bunch with inheritance. However, none of this is a game breaker and for those of you who have a good workable flow with this approach and are happy with it, please continue to use it. I however, am not happy with the flow, also BDD really is not easy for me to teach. Some of you may get it easily and teach it easily, but .Net developers as a whole seem to be driven towards framework specific knowledge (telling them to not think “test” when test is staring at them on the method messes with their heads), and even then it’d better not be too “cutting edge” in language features or friction becomes a risk where someone may end up learning more than just BDD.

So what are my goals then for SpecMaker?

It has to be terse, and be shorter than the NUnit approach.
Eliminate underscores where possible.
Act as a series of guidelines to beginning BDD’ers.
Output spec results in a variety of formats.
Remove the “Test” entirely from the vocabulary.
Use convention over configuration where possible.
Provide custom matchers and a DSL to make your own custom matchers.

What will it not do or be?

Do acceptance testing. This could change in the future, but then I’d be inclined to borrow code from others that do this well already.
Make BDD a “click next” thing. There is no substitute for continually trying to refine your approach and understand “behavior” and what that means. I’m not sure there is one true right answer here and that makes it so incredibly hard to make a framework that does that.
Be as nice as RSpec.
Integration tests. This is setup very much one-to-one. Again this was a simplicity choice and as I dog food this I may completely change my mind here.

So with all the ceremony out of the way here is where I’m at so far for the same BDD code above only with SpecMaker

 public class GameSpec : BaseSpec

    {

        private void startGame(){

          //left out for clarity

        }  

        public void when_starting_a_game()

        {

            startGame();

            should("have 3 lives", ()=>

                    {

                       _game.lives.Has(3).Total();

                    }

                );

            should("require valid username", RuleIs.Pending);

        }

     }

Running specmaker.exe on the the dll where this spec is located outputs something like this:

At this time specs come from the class name minus “Spec” at the end. Context start with “when” or the method will not get picked up as context. This is also staggeringly new and may have bugs, issues, bad docs, ugly code, etc. But since I’ve moved this to Github I encourage everyone to have a go and improve it as they see fit or for their own purposes. I have a number of plans and ideas to improve this, but I feel this is a good start to get some positive work done and save myself some grief over my NUnit based tests.

Let me know what you think, any and all criticism is appreciated, but at the end of the day this does actually fulfill an itch that I myself have had for some time.

Windsor style convention over configuration in StructureMap

Ryan Svihla — Fri, 07 Aug 2009 12:47:00 GMT

So I was trying to get my BDD project SpecMaker working on Mono only to find out my usual IoC container of choice Castle Windsor does not play nice. StructureMap I’ve heard plays nice in Mono, and I know a bunch of people that use it, so I gave it a whirl.

My original Windsor configuration looked something like this:

 var kernel = new DefaultKernel();

 kernel.Register(Component.For<IReportFactory>().ImplementedBy(factory)); //load implementation

 kernel.Register(

                AllTypes.Pick().

                FromAssembly(GetType().Assembly).

                WithService.FirstInterface().  //regardless of naming first interface is listed as service

                Configure(c=>c.LifeStyle.Transient));

I changed this to the what I thought was the StructureMap equivalent:

 var container = new Container(registry =>

            {

                registry.Scan(x =>

                {

                    x.TheCallingAssembly();

                    x.With<DefaultConventionScanner>(); //does NOT configure the first service interface

                });

            });

container.SetDefault(typeof(IReportFactory), new ConfiguredInstance(factory)); //specific implemenation

Ok so I ran my code and nearly none of my classes were registered only interfaces and classes that were in a pattern like ISpecFinder/SpecFinder. Looking at SM’s site this is actually expected behavior, and I was not able to find anything that would work the same way as I was used to. I’m not personally a fan of naming interfaces exactly like class names, so after looking around a bit at my options with SM and a brief perusal of Windsor’s source, I used StructureMap’s extensibility to come up with the following:

class FirstInterfaceConvention : TypeRules,ITypeScanner

    {

        public void Process(Type type, PluginGraph graph)

        {

            if (!IsConcrete(type)) return;  //only works on concrete types

            var firstinterface = type.GetInterfaces().FirstOrDefault(); //grabs first interface

            if(firstinterface!=null)

                graph.AddType(firstinterface, type); //registers type

            else

            {

                graph.AddType(type); //adds concrete types with no interfaces

            }

        }

    }

    public class IoCConfig

    {

        public virtual ISpecRunner InitializeRunner(Type factory)

        {

            var container = new Container(x=> x.Scan(

                                                   y=>

                                                       {

                                                           y.TheCallingAssembly();

                                                           y.With<FirstInterfaceConvention>();

                                                       }

                                                   ));

            container.SetDefault(typeof(IReportFactory), new ConfiguredInstance(factory));

            return container.GetInstance<ISpecRunner>();

        }

    }

and perfecto! Everything now works as it did before and I was happy that something that wasn’t supported out of the box was so very simple to add myself. Finally, SpecMaker now appears to be working in mono and is nearing a release for me which I will share shortly.

Discussion: Mentoring, Knowledge Transfer and Different Styles Of Learning

Ryan Svihla — Fri, 17 Jul 2009 00:06:00 GMT

I’ve been very fortunate that for most of my adult life, even before I got into tech I’ve been “the boss”, and had gobs of mentoring opportunities. I enjoy it and more importantly I know I will have a greater effect for my company's growth through teaching than I would by solely focusing on my advancement.

My students have been visual learners, “doing” learners, audio learners, those that learn by rote memorization with photographic memories, “theorists” that are fascinated by the theory of things far more than the practice, and those that can’t memorize anything unless they see an entire system in place. Each type or combination needs to have information presented in different ways.

I don’t even remotely pretend to know MOST of the answers here, in fact I’ll frankly admit there are just some types of people I’ve mentored unsuccessfully despite much effort on my part and trying different tactics over many years.

The best results have come with any “system” based learners (theoretical or practical), particularly those of a “doing” bent. However, that is less me and more them. People like that just need to be thrown into a situation and given small amounts of guidance, encouragement, a gob of reference material (not tutorials, reference only!) and they take off and explode. Really, you just need to give people like these work to do and get out of their way so they feel free to grow.

Visual rote learners, however are the end of me. I’ve had several work for me at this point and I’m at a loss . I can draw diagrams all day long for them, show them examples, try and give them a number of examples themselves to do, and the minute they get one step off track they’re completely lost. I’ve tried going into great depth systems wise, answering and demoing any questions they have, teach them the basics of deductive reasoning and troubleshooting, on and on and on. On average I spend 5 times as long with them and I never get anywhere near the success level I do with other types of learners.

Now I’m beginning to believe I’m just on a fools errand. If there is a way to inspire visual rote learners into becoming dynamic problem solvers, I certainly do not have the skill set to do so.

Anyone out there have any particular success mentoring those that don’t learn by “trying” things out, and don’t have to understand a whole complete system to begin work? What tactics did you use? Did you just assume a slower trajectory but take them through the same material as the faster students? Did you have to resort to AgileJoe like Jedi Mind tricks?

I’m convinced there is a way to reach the vast majority of "struggling" technical people and turn them into at least respectable problem solvers and adequate developers/techs/whatever. I had my own struggles with learning until I got into my mid 20’s, and I was able to figure out what worked for me. Pains me to see others in the same boat I was in and I’ve so far been unable to help a decent percentage of my past students.

Lessons From My First Attempt At Bringing Agile Into A Non-Agile shop.

Ryan Svihla — Sun, 12 Jul 2009 07:06:08 GMT

Since I’m moving on from my current employer I figure now is a great time to share my post-mortem of my time there, specifically what worked, what didn’t and what did I learn.

Iterations Of Any Length

We tried 3 weeks, 2 weeks, 1 week, monthly and eventually just sort of tossed the system out all together as our team got shrunk from 5 to 2. We did find get a fairly desirable result ultimately but more on that later. Ultimately our problem was especially as we got smaller, dealing with our “particular” realities made iterations tend to fall apart, we were spending a fair amount of time trying to torture ever more atomic stories together. Cutting iterations out all together and focusing on features being completed was MUCH more time effective for us.

Continuous Integration

We had 3 different build servers at various points. All worked well, and our scripts were build server agnostic enough to not take long to switch over. There was a bit of a learning curve with some projects and the build system (modular dll’s and app domains especially). While in the end this worked well for our purposes, I failed in sharing the knowledge here, and never figured out how to get other members of the team to get up to speed on the entire build server concept .

Note: oddly no “build applet” was nearly as effective at letting people know the build failed as sending them an email automatically on failure.

TDD, BDD And Unit Testing

Very mixed bag here. I was successful in getting the concept of unit testing in, but not TDD. However, our tests still are not totally flexible, and the training to do that is significant (creates dependencies on a fair amount of knowledge not specifically related to testing). For my part it honestly took me a year to be able to write a test that was not brittle.

I’m still a bit stumped how to get a typically trained Microsoft developer team to really dive into the concept of TDD, “writing the code you want” was met with blank stares. Data driven mindset was maintained to some degree throughout my time there.

BDD took me awhile to get something I felt comfortable with so I don’t think I trained with the proper approach. I emphasized specification and example a lot and for some reason this lead to everyone striving for functional tests.

Finally, handing customers spec results was somewhat helpful. Still had some things “be fine” then “that’s not what I meant”, but it still cut down the amount of misunderstanding.

SOLID Principles

This was the easiest thing overall. DIP and IoC was resisted heavily at first, but it was not much effort to come up with scenarios where it was patently obvious how much more flexible code became. I focused a ton of “different data sources” since we have a number of api’s we deal with that gave me the buy-in I needed.

SRP took a bit but once they got it they got it big (almost too well). Ultimately, I think the dev’s enjoyed having “another box to check off” as they finished each class. Resharper training also helped cut down the time to make classes and methods I believed this also had a role in SRP adoption.

OCP came with focusing on IoC and DiP so heavily.

LSP only came up occasionally since we tended to have a lot of small classes.

YAGNI And DRY

YAGNI - Always fought this myself, and correspondingly my team struggled with both as well. YAGNI by the end started to work just because I began stripping out all extra requirements full stop, and immediately stopping any development when “extra” features were added out of the blue by an enthusiastic developer. This took a heavy hand and vigilance, and convincing was less the issue than fighting habits developed over years.

DRY – Was especially rampant in our test code, and we had our bits of duplication across projects and different developers. Like our YAGNI violations I think this was years of habits holding us back.

Automated Acceptance Tests

Never got close. Not enough stakeholders were interested, the training on acceptance tests ran into the same problem I had with build scripts that no one understood systems enough to make this function properly unless I was involved heavily, and then breaks also required my involvement. Finally scrapped them and moved to specs and manual acceptance testing by the stakeholders that cared about it in the way that they cared about it.

Release Frequently Release Often

The more we released the better it worked for us. For us there was a clear link between stakeholder satisfaction and frequent releases (10+ some weeks). I have a few theories about this:

We had stakeholders with various levels of comfort on their subject matter. Domain experts that knew everything about their domain did not need as much “final product” feedback as some others, who really were only happy when they saw the end product after every request.
Most of our projects had few moving parts. The tendency of past development seems to have been drawn to complexity, even when the raw domain didn’t need it. Simple projects can be released more often when less moving parts are there to break.
Our stakeholders seemed to be far more results driven than UI driven. We seemed to attract a lot of process driven people who were content with having a lousy limited UI as long as the file went out in the correct format. We could again release more often with a stubbed in UI without any real penalty, as ultimately they just wanted at least one front to back business case working.

Summary

There is more that I’m sure that I’m missing that may be useful. In the end, I was disappointed in my attempt to move the shop to an Agile concept only because I was not able to spread the knowledge effectively. My inexperience at the time in Agile myself made it rougher on everyone else trying to learn from me.

If I had to do it over again, I’d focus on SOLID, YAGNI, DRY and training a systems administrator in build scripts (it’s just a hard skill set for the average MS developer), and of course release often. If I had focused on these things first the immediate dividends would have been far larger. Once they had the necessary engineering credentials, TDD would have been less of a time sink, and maybe we’d have had the discipline for Iterations. Someday I hope to have the opportunity to build an Agile team from scratch again. I’m grateful to my employer and my co-workers who gave me the opportunity to try and advance things forward (which we did, just not as far as I would have liked).