MySql crossplatform tip (especially with an ORM)

This is just a small post for my own reference and who knows it might even save someone some work.

I am developing on windows and use Subsonic as an ORM in a current project. At the client site we have a linux server running MySql. We however develop against a MySql on windows. One particular pain point we encountered was that by default on windows all identifiers, including table and column names are transformed in lower case. We got a lot of table not found exceptions when we tried  (and failed) to deploy our newly generated ORM classes at the customer site.

The solution is very simple and the behavior well documented on the MySql site.

In your mysql installation folder (by default: C:\Program Files\MySQL\MySQL Server 5.5) add the following line in the my.ini under the [mysqld] section (which configures the mysql service):

lower_case_table_names=0

After restarting the service your MySql database will have the same identifier casing behavior as a Linux installation.

Tags:

Solve: “Could not load VCBuild.exe” on a CI (build server)

When I came across the following error, I needed to search a bit before I found the solution. I hope this will help someone facing the same issue.

The error I faced was:

error MSB3411: Could not load the Visual C++ component "VCBuild.exe". If the component is not installed, either 1) install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5, or 2) install Microsoft Visual Studio 2008

The following blog post of Jeremy Jameson actually gave me a clue on how to solve this.

You do not want to install visual studio on your build server. Keep it as clean as possible to prevent the “It only works on my machine”-issues. Installing the windows SDK isn’t ideal either but acceptable.

After installing the right components, the next step is to include VcBuild in the path. You can configure it in your CI but I think changing the global path environment variable is more acceptable. To set the global path variable: right click My Computer, choose Properties. In Windows 7 click on Advanced system settings.

image In this new window click on Environment Variables.

image

In the Environment Variables dialog choose Path under System variables. Click on Edit.

image Add the following path for x86 systems (It might look differently for x64 systems):

;C:\Program Files\Microsoft Visual Studio 9.0\VC\vcpackages

image Make sure the path is seperated from the previous path with a semicolon (;). And yes even when you install the windows sdk, VcBuild.exe is located in the Microsoft visual studio folder instead of a windows sdk something.

Press Ok, Ok, Ok and Close :) all windows should be gone now.

One last catch with : restart it to have the new path availlable during the build steps.

Tags: , , , , ,

2D automagically becomes 3D

I never thought I would be posting something else but coding stuff… But this one’s definitely worth a post. It could be the next generation of …. loved it. You can actually try it out yourself on the web using a java applet on this page. It definitely shows how important a good user experience is.

I actually found it through this pixel in gene blog

Tags: , , , ,

Integrating nBehave with nUnit, MbUnit or Gallio

On a lot of testing frameworks/runners I read   integration. But I found no examples. But then it just came to me it’s actually very very simple. So after banging my head against the wall a few times and then a few times more for being stupid, I thought I should share this with others. Please let me know that you found this helpful so I feel less stupid.

Let’s start with a specification example for a repository service:

[Theme("Storing and Retrieving Data Objects from the repository.")]
public class RepositorySpecs
{
    [Story]
    public void StoreObjectInRepository()
    {
        Story storeObjectStory = 
		new Story("Store and retrieve an object in the repository");

        storeObjectStory.AsA("service in the platform")
                        .IWant("to persist data")
                        .SoThat("I can retrieve this later");

        IRepository repository = null;
        object dto = null;

        storeObjectStory.WithScenario("a flat data object whitout IStorable")
            .Given("the data object is a $type",
		"singleLevelDto",
		dtoType => { dto = DtoFactory.GenerateDto(dtoType); })
            .And("a IRepository service",
		() => repository = Service.Get<IRepository>())
            .When("I store the object under $id and the default datastore",
		"SimpleDtoTest",
		objectId => { repository.Store(string.Empty, objectId, dto); })
            .Then("I should be able to retrieve it by supplying $id and the type",
		"SimpleDtoTest",
		objectId => { repository
		    .Retrieve<SingleLevelSimpleDto>(string.Empty, objectId)
		    .PropertyValuesAreEqual(dto); });
    }
}

We can easily change the specification in an test and still keep the specification descriptive format. Let’s start by adding the following using statements to the top of the file:

using That = NUnit.Framework.TestAttribute;
using Should = NUnit.Framework.DescriptionAttribute;
using Context = NUnit.Framework.TestFixtureAttribute;
using Specification = NUnit.Framework.TestAttribute;
using Concerning = NUnit.Framework.CategoryAttribute;

After using these attributes our example now looks like this:

[Theme("Storing and Retrieving Data Objects from the repository."),
 Context,
 Concerning("IRepository")]
public class RepositorySpecs
{
    [Story, That, Should("Store an Object in the repository")]
    public void StoreObjectInRepository() 
    { 
        Story storeObjectStory = new Story("Store and retrieve an object in the repository"); 

        storeObjectStory.AsA("service in the platform") 

This is now a fully functioning and valid NUnit test, recognized by resharper (if you’re not using this tool you are really missing out on productivity).

resharper example

Of course NUnit recognizes it too:

image

There are still two things that bug me:

  • I am duplicating the text in the Should attribute and the Story constructor.
  • you miss out on a lot of the BDD output. But that’s what dryrun is for I suppose.

But for now I’ll be able to live with both since I can easily debug from within visual studio (I know I am easy to please :) )

Tags: , , , , , , ,

BDD with nbehave tips

While using nBehave on projects especially those where we collaborated with external partners I created/shared the following list of guidelines/tips which might save you some time too. So here I am sharing it to a broader audience… maybe :)

1. All declarations/instantiations must be placed after the narrative
2. All code that can fail must be inside an actionstep delegate or lambda
3. How to execute and debug using nBehave Console Runner
4. Improving execution and debugging by integrating with nUnit

All declarations/instantiations must be placed after the narrative

The first two solve the same problem. When code fails that’s not inside an actionstep it is unhandled (especially with the nbehaverunner, look at 4 for a different runner) and the whole thing just stops with an exception. What’s even worse invalid/incomplete xml is written that will break any xslt transformation or processing which I found out the hard way while maintaining a CI (). How to integrate nBehave (or rather how I integrated it) with Hudson is worth a post on it’s ownn that will follow.

uses the Story class to execute and listen to output of the specification. A very important consequence is that if code that is not part of an actionstep (inside a lambda or delegate as a parameter of an actionstep) the complete test fails with no or invalid output.

A lot of us would write a spec in this way (since we are used to put our declarations at the beginning of a code block:

IRepository repository = Service.Get<IRepository>();

Story storeObjectStory = new Story("Store and retrieve an object in the repository");

When the Service.Get fails the story will never be created and we have no output on why or which test failed.

Our first guideline is to move all declarations past the story narrative. We created a modified nBehave runner build that produces valid output even when the code throws an exception (if you’re interested let me know).

Let's look at our modified example:

Story storeObjectStory = new Story("Store and retrieve an object in the repository");

storeObjectStory.AsA("service in the platform")
                                .IWant("to persist data")
                                .SoThat("I can retrieve this later");

IRepository repository = Service.Get<IRepository>();

storeObjectStory.WithScenario("the single level data object does not implement IStorable")

All code that can fail must be inside an actionstep delegate or lambda

nBehave will nicely catch and report exceptions in code that are part of an actionstep. Code failing outside an actionstep will result in unexpected behaviour.

To improve our previous example we should place the initialization of the repository component inside an actionstep. Since it is part of the setup the appropriate place is the “Given”-actionstep (or a subsequent “And”-actionstep).

Story storeObjectStory = new Story("Store and retrieve an object in the repository");

storeObjectStory.AsA("service in the platform")
                                .IWant("to persist data")
                                .SoThat("I can retrieve this later");

IRepository repository = null;
object dto = null;

storeObjectStory.WithScenario("the single level data object does not implement IStorable")
        .Given("the data object is a $type", "singleLevelDto", dtoType => { dto = DtoFactory.GenerateDto(dtoType); })
        .And("a IRepository service", () => repository = Service.Get<IRepository>())
        .When("I store the object under $id and the default datastore", "SimpleDtoTest", objectId => { repository.Store(string.Empty, objectId, dto); })
        .Then("I should be able to retrieve it by supplying $id and the type", "SimpleDtoTest", objectId => { repository.Retrieve<SingleLevelSimpleDto>(string.Empty, objectId).PropertyValuesAreEqual(dto); });

 

How to execute and debug using nBehave Console Runner

A major pain point of nBehave is the difficulty of testing it on a local machine. One way of executing these is by creating or re-using the batch scripts used by your CI-server (in our case it’s hudson). We use a tools/scripts folder in our source tree.

We have a nBehaveRunAll.cmd which will execute all component specs. It accepts an output directory parameter relative to the different project folders. By default it uses the output of the debug (anycpu) configuarion (the configuration chosen in visual studio). The following example shows how to specify the release configuration:

... >Tools\Scripts\nBehaveRunAll.cmd bin\Release 

To debug you can specify the nbehave console runner in your project properties:

image

In Start External Program fill in the path to the NBehave console runner.

The command line arguments consist of 2 arguments:

  • The assembly you want to run
  • xml=<xml output> parameter. This one is important. Without it you are not executing the tests.

Now you can set breakpoints and start debugging by pressing F5 :)

Keep in mind that these settings are configuration dependent! This means that whenever you change the build configuration (for instance to release or x86 instead of any cpu) you’ll need to reenter these values. Is there no better way you ask? Read on to find out how…

Improving execution and debugging by integrating with NUnit

The above is fine if you are a build server but a lot of developers use resharper or the (or the Galio Icarus) GUI to run and debug tests. Changing my specs into this made such an impact on the usage of NBehave with our partners, that I created a seperate blog post explaining this. Before a lot of the partners “forgot” to execute, update or correct the executable specifications (BDD tests) because they perceived them as being hard and time consuming. By changing them into NUnit tests this problem was solved.

Tags: , , , , , , , ,

Fully Trust a share in .Net 2.0, 3.0 and 3.5

Just some quick info for those struggling with the same problems.

When i google for this almost all answers predate the .net 3.0 framework. Everybody knows that 3.0 are a set of libraries that extend 2.0. what i missed was that 3.5 uses the same core assemblies as 2.0 including the same security settings. This means you still use the CasPol installed from the 2.0 to set full trust in 3.5. Be aware thought that you’ll need to run it for the 32-bit version and 64-bit version separately.

There is a nice post on how to do this from the .net security blog from which I included the next fragment:

since network shares by default only get localintranet permissions, it's relatively common to want to use CasPol to fully trust some shares that you control and know are safe.  However, CasPol syntax being what it is, the command to do this isn't immediately obvious.  if i wanted to trust everything on the share \\shawnfa-srv\tools, the command:

caspol.exe -m -ag 1.2 -url file://\\ShawnFa-Srv/Tools/* FullTrust

would setup the policy to do what i needed.  lets break down this command:

  • -m  - modify the machine level of the policy.  this is needed, since the machine level is where all of the default policy lives.  on nt platforms it's also the default level that CasPol works with, however on Win9x, CasPol will default to the user level, so putting -m in the command line explicitly tells CasPol to use the correct level.
  • -ag 1.2  - add a code group under group 1.2.  in the default policy, group 1.2 is the localintranet group, so the new code group that we're creating will only be checked if the file comes from the intranet.
  • -url file://\\ShawnFa-Srv/Tools/- The membership condition for the new code group should be a UrlMembershipCondition, and it should match anything with a URL that starts with file://ShawnFa-Srv/Tools, meaning that any file on the \\ShawnFa-Srv\Tools share will match this code group.
  • fulltrust  - the permission set to grant assemblies that match the code group.  in this case, fulltrust.

Executing the above command will give you a yes/no confirmation prompt which you can suppress by adding the –pp off switch. Now it’s perfectly usable in scripts.

I kept wondering about 2 things:

  • what if the mapped folder was included in the trusted group?
  • where does it store these settings?

Get the code groups.

To get the right -ag argument for the trusted group call:

caspol.exe –lg

which will give you something like this:

Microsoft (R) .NET Framework CasPol 2.0.50727.3053
Copyright (c) Microsoft Corporation.  All rights reserved.

Security is ON
Execution checking is ON
Policy change prompt is ON

Level = Machine

Code Groups:

1.  All code: Nothing
   1.1.  Zone - MyComputer: FullTrust
      1.1.1.  StrongName - 002400000...: FullTrust
      1.1.2.  StrongName - 000000000...: FullTrust
   1.2.  Zone - Intranet: LocalIntranet
      1.2.1.  All code: Same site Web
      1.2.2.  All code: Same directory FileIO - 'Read, PathDiscovery'
   1.3.  Zone - Internet: Internet
      1.3.1.  All code: Same site Web
   1.4.  Zone - Untrusted: Nothing
   1.5.  Zone - Trusted: Internet
      1.5.1.  All code: Same site Web
Success

From this output we now know that we need to use 1.5 as an –ag argument to FullTrust a path from the Trusted group.

Where is it stored?

The security information is stored in the security.config file (duh) in the following location:

%WINDIR%\Microsoft.NET\Framework\v2.0.50727\CONFIG

Our example inserts the following xml under the Trusted codegroup element:

<CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust">
    <IMembershipCondition class="UrlMembershipCondition" version="1" Url="file://\\ShawnFa-Srv/Tools/*"/>
</CodeGroup>

If you don’t want the settings removed by a call to:

CasPol -all –reset

You can copy the security.config to security.config.default

Tags: , , , , , , , , ,

Detect if an assembly is a managed assembly

While using a great IOC/DI container (read for those not familiar with it read this post) I wrote some code to auto detect and auto load the necessary dependencies (I really, really, REALLY hate the XML configuration file so …) Besides a lot of other issues with this method worth a blog post on it’s own, the whole thing crashed as soon as an unmanaged assembly was found in the path.

After some googling I found a few ways to solve my problem:

Use .Net Reflection:

Try to do a AssemblyName.GetAssemblyName(path); If it throws an exception it’s not a managed assembly. As you would suspect this is horribly slow.

Read the binary data from the assembly to detect the assembly type.

Actually the .Net managed assemblies are with specific values in the header. To read these we could use the command and then walk through the headers with pointers as explained on this site. Since this loads the assembly in memory it was a no-go for me.

There are alot of unmanaged solutions out there (read here and here) that are candidates to port to c# but Rupreet’s weblog saved the day. And here are the relevant code fragments of my AssemblyInfo class:

   1:  try
   2:  {
   3:      if (dataDictionaryRVA == null)
   4:      {
   5:          GetHeaders();
   6:      }
   7:      return dataDictionaryRVA[14] != 0;
   8:  }
   9:  catch (Exception)
  10:  {
  11:      // when an error occurs return false.
  12:  }
  13:  return false;

Obviously the interesting code is inside the GetHeaders method including the comments from Rupreet:

   1: dataDictionaryRVA = new uint[16];
   2: dataDictionarySize = new uint[16];
   3:  
   4: using (FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read))
   5: {
   6:     BinaryReader reader = new BinaryReader(fs);
   7:  
   8:     //PE Header starts @ 0x3C (60). Its a 4 byte header.
   9:     fs.Position = PeHeaderStart;
  10:     peHeader = reader.ReadUInt32();
  11:  
  12:     //Moving to PE Header start location...
  13:     fs.Position = peHeader;
  14:     peHeaderSignature = reader.ReadUInt32();
  15:  
  16:     //We can also show all these value, but we will be       
  17:     //limiting to the CLI header test.
  18:     machine = reader.ReadUInt16();
  19:     sections = reader.ReadUInt16();
  20:     timestamp = reader.ReadUInt32();
  21:     pSymbolTable = reader.ReadUInt32();
  22:     noOfSymbol = reader.ReadUInt32();
  23:     optionalHeaderSize = reader.ReadUInt16();
  24:     characteristics = reader.ReadUInt16();
  25:  
  26:     /*
  27:     Now we are at the end of the PE Header and from here, the
  28:     PE Optional Headers starts...
  29:     To go directly to the datadictionary, we'll increase the      
  30:     stream’s current position to with 96 (0x60). 96 because,
  31:     28 for Standard fields
  32:     68 for NT-specific fields
  33:     From here DataDictionary starts...and its of total 128 bytes. DataDictionay has 16 directories in total,
  34:     doing simple maths 128/16 = 8.
  35:     So each directory is of 8 bytes.
  36:     In this 8 bytes, 4 bytes is of RVA and 4 bytes of Size.
  37: 
  38:     btw, the 15th directory consist of CLR header! if its 0, its not a CLR file :)
  39:     */
  40:  
  41:     dataDictionaryStart = Convert.ToUInt16(Convert.ToUInt16(fs.Position) + 0x60);
  42:     fs.Position = dataDictionaryStart;
  43:  
  44:     for (int i = 0; i < 15; i++)
  45:     {
  46:         dataDictionaryRVA[i] = reader.ReadUInt32();
  47:         dataDictionarySize[i] = reader.ReadUInt32();
  48:     }
  49:  
  50:     fs.Close();
  51: }

My complete class implementation is here:

   1: using System;
   2: using System.IO;
   3: using System.Reflection;
   4:  
   5: namespace Williame.Koen.Blog
   6: {
   7:     /// <summary>
   8:     /// Returns information about an assembly without loading it into the appdomain.
   9:     /// </summary>
  10:     public class AssemblyInfo
  11:     {
  12:         /// <summary>
  13:         /// PE Header starts @ 0x3C (60). Its a 4 byte header.
  14:         /// </summary>
  15:         public const long PeHeaderStart = 0x3C;
  16:  
  17:         protected uint peHeader;
  18:         protected uint peHeaderSignature;
  19:         protected ushort machine;
  20:         protected ushort sections;
  21:         protected uint timestamp;
  22:         protected uint pSymbolTable;
  23:         protected uint noOfSymbol;
  24:         protected ushort optionalHeaderSize;
  25:         protected ushort characteristics;
  26:         protected ushort dataDictionaryStart;
  27:         protected uint[] dataDictionaryRVA;
  28:         protected uint[] dataDictionarySize;
  29:         protected string file;
  30:         protected AssemblyName name; 
  31:  
  32:         public AssemblyInfo(string fileName)
  33:         {
  34:             this.file = Path.GetFullPath(fileName);
  35:         }
  36:  
  37:         protected void GetHeaders()
  38:         {
  39:             dataDictionaryRVA = new uint[16];
  40:             dataDictionarySize = new uint[16];
  41:  
  42:             using (FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read))
  43:             {
  44:                 BinaryReader reader = new BinaryReader(fs);
  45:  
  46:                 //PE Header starts @ 0x3C (60). Its a 4 byte header.
  47:                 fs.Position = PeHeaderStart;
  48:                 peHeader = reader.ReadUInt32();
  49:  
  50:                 //Moving to PE Header start location...
  51:                 fs.Position = peHeader;
  52:                 peHeaderSignature = reader.ReadUInt32();
  53:  
  54:                 //We can also show all these value, but we will be       
  55:                 //limiting to the CLI header test.
  56:                 machine = reader.ReadUInt16();
  57:                 sections = reader.ReadUInt16();
  58:                 timestamp = reader.ReadUInt32();
  59:                 pSymbolTable = reader.ReadUInt32();
  60:                 noOfSymbol = reader.ReadUInt32();
  61:                 optionalHeaderSize = reader.ReadUInt16();
  62:                 characteristics = reader.ReadUInt16();
  63:  
  64:                 /*
  65:                 Now we are at the end of the PE Header and from here, the
  66:                 PE Optional Headers starts...
  67:                 To go directly to the datadictionary, we'll increase the      
  68:                 stream’s current position to with 96 (0x60). 96 because,
  69:                 28 for Standard fields
  70:                 68 for NT-specific fields
  71:                 From here DataDictionary starts...and its of total 128 bytes. DataDictionay has 16 directories in total,
  72:                 doing simple maths 128/16 = 8.
  73:                 So each directory is of 8 bytes.
  74:                 In this 8 bytes, 4 bytes is of RVA and 4 bytes of Size.
  75: 
  76:                 btw, the 15th directory consist of CLR header! if its 0, its not a CLR file :)
  77:                 */
  78:  
  79:                 dataDictionaryStart = Convert.ToUInt16(Convert.ToUInt16(fs.Position) + 0x60);
  80:                 fs.Position = dataDictionaryStart;
  81:  
  82:                 for (int i = 0; i < 15; i++)
  83:                 {
  84:                     dataDictionaryRVA[i] = reader.ReadUInt32();
  85:                     dataDictionarySize[i] = reader.ReadUInt32();
  86:                 }
  87:  
  88:                 fs.Close();
  89:             }
  90:         }
  91:  
  92:         public bool IsManaged
  93:         {
  94:             get
  95:             {
  96:                 try
  97:                 {
  98:                     if (dataDictionaryRVA == null)
  99:                     {
 100:                         GetHeaders();
 101:                     }
 102:                     return dataDictionaryRVA[14] != 0;
 103:                 }
 104:                 catch (Exception)
 105:                 {
 106:                     // when an error occurs return false.
 107:                 }
 108:                 return false;
 109:             }
 110:         }
 111:  
 112:         public AssemblyName Name
 113:         {
 114:             get
 115:             {
 116:                 if (name == null)
 117:                 {
 118:                     name = AssemblyName.GetAssemblyName(file);
 119:                 }
 120:                 return name;
 121:             }
 122:         }
 123:     }
 124: }

As you can see, I also included the AssemblyName which can give me some more information withtout loading the assembly in memory.

Issues:

  • This is example code so be careful with it in production code.
  • In addition, check the "magic" field of the PE header (the first 2 bytes) to ensure it is a 32-bit image file (0x010B). The data directory table for 64-bit PE headers start at offset 112 - as opposed to 96 for 32-bit headers.
  • This example ignores recommendations in Microsoft's PE specification to check the values of size fields so you don't read the wrong data

Tags: , , , , , ,

My first blog post :)

I finally did it… I started my blog. (small victory dance by yours sincerely).

I am currently trying out Windows Live Writer and BlogEngine.Net and am loving them.

Now after the obligatory blabbering in this typical first post. I promise to blog about some more interesting things. Programming, programming, and of course software development :).

Tags: , ,