It all started with this article on CodeProject from April last year: https://www.codeproject.com/articles/1083348/csharp-bad-practices-learn-how-to-make-a-good-code

The author shows a piece of badly written and then goes through a slew of refactorings following Liskov, strategy patterms, dependency injection and many other well-known principles.

While I’m a big fan of good software practices, the number 1 principle I like to adhere to is KISS: (Keep it simple, stupid). If you have been reading my blog before, you’ll certainly have come across the theme. I’m not the only though, there were a few follow up posts as well:

• Don’t let cleaning go overboard (by Ralf Westpahl)
• An F# rewrite (by Roman Bassart)
• Using C# 7 and Succin<T> to give F# a run for its money (by David Arno)

I like many of the ideas expressed in the above posts, but I couldn’t stop to think about how this code could be made much simpler. Looking at the posts I still see things that are complicating matters much more than necessary.

Code before refactoring

For reference, this is the initial code from the first post:

public class Class1
{
  public decimal Calculate(decimal amount, int type, int years)
  {
    decimal result = 0;
    decimal disc = (years > 5) ? (decimal)5/100 : (decimal)years/100; 
    if (type == 1)
    {
      result = amount;
    }
    else if (type == 2)
    {
      result = (amount - (0.1m * amount)) - disc * (amount - (0.1m * amount));
    }
    else if (type == 3)
    {
      result = (0.7m * amount) - disc * (0.7m * amount);
    }
    else if (type == 4)
    {
      result = (amount - (0.5m * amount)) - disc * (amount - (0.5m * amount));
    }
    return result;
  }
}

This is indeed not very nice code. What I do like about it though, is that it’s compact. When I read this, I can probably figure out relatively quickly what this code does. It has many problems though (as discussed in the original post).

The “issue” I have with the refactorings in the other posts is that they try to cater for use cases that simply aren’t specified. From what I can tell, these are the requirements:

• Give a discount based on what type of customer it is
• Give a loyalty discount which equals the amount of years the customer has been active, with a maximum of 5
• Both discounts are cumulable

The simplest possible solution

UPDATE: after comments on twitter / reddit, I noticed that the tests were incorrect. I had taken them from one of the refactorings and assumed they were correct. I have modified the data and updated the tests to reflect what the original code does.

UPDATE 2: I went against my own adage: going to far. Smuggling the discount info into the enum was too much. I have refactored it to a dictionary, which is easier to maintain.

static readonly Dictionary<Status, int> Discounts = new Dictionary<Status, int>
{
    {Status.NotRegistered, 0 },
    {Status.SimpleCustomer, 10 },
    {Status.ValuableCustomer, 30 },
    {Status.MostValuableCustomer, 50 }
};
decimal applyDiscount(decimal price, Status accountStatus, int timeOfHavingAccountInYears)
{
    
    price = price - Discounts[accountStatus] * price/100;
    return price - Math.Min(timeOfHavingAccountInYears, 5)* price/100;
}

It’s a simple calculation, so why not express is at as a simple calculation? While I see the power of functional programming, sometimes discriminated unions, partial application and the likes are just overkill for simple problems.

For a coding kata like this, I understand that people want to provide a most elegant solution for future needs, but unfortunately I see these patterns arise far too often, complicating already complex problems even more.

Bonus 1: Tests

[Fact]
public void Tests()
{
    applyDiscount(100m, Status.MostValuableCustomer, 1).Should().Be(49.5000m);
    applyDiscount(100m, Status.ValuableCustomer, 6).Should().Be(66.5000m);
    applyDiscount(100m, Status.SimpleCustomer, 1).Should().Be(89.1000m);
    applyDiscount(100m, Status.NotRegistered, 0).Should().Be(100.0m);
}

Bonus 2:

Want a UI with that? Here you go!

The post Refactoring taken too far appeared first on Kenneth Truyers.

The reason why it says stupid in the man-pages is that it makes no assumptions about what content you store in it. The underlying git model is rather basic. In this post I want to explore the possibilities of using git as a NoSQL database (a key-value store). You could use the file system as a data store and then use git add and git commit to save your files:

# saving a document
echo '{"id": 1, "name": "kenneth"}' > 1.json
git add 1.json
git commit -m "added a file"

# reading a document
git show master:1.json
=> {"id": 1, "name": "kenneth"}

That works, but you’re now using the file system as a database: paths are the keys, values are whatever you store in them. There are a few disadvantages:

• We need to write all our data to disk before we can save them into git
• We’re saving data multiple times
• File storage is not deduplicated and we lose the benefit git provides us for automatic data deduplication
• If we want to work on multiple branches at the same time, we need multiple checked out directories

What we want rather is a bare repository, one where none of the files exist in the file system, but only in the git database. Let’s have a look at git’s data model and the plumbing commands to make this work.

Git as a NoSQL database

Git is a content-addressable file system. This means that it’s a simple key-value store. Whenever you insert content into it, it will give you back a key to retrieve that content later.
Let’s create some content:

#Initialize a repository
mkdir MyRepo
cd MyRepo
git init

# Save some content
echo {"id": 1, "name": "kenneth"} | git hash-object -w --stdin
da95f8264a0ffe3df10e94eed6371ea83aee9a4d

Hash-object is a git plumbing command which takes content, stores is it in the database and returns the key

The –w switch tells it to store the content, otherwise it would just calculate the hash. the –-stdin switch tells git to read the content from the input, instead of from a file.

The key it returns is a sha-1 based on the content. If you run the above commands on your machine, you’ll see it returns the exact same sha-1. Now that we have some content in the database, we can read it back:

git cat-file -p da95f8264a0ffe3df10e94eed6371ea83aee9a4d
{"id": 1, "name": "kenneth"}

Git Blobs

We now have a key-value store with one object, a blob:

There’s only one problem: we can’t update this, because if we update the content, the key will change. That would mean that for every version of our file, we’d have to remember a different key. What we want instead, is to specify our own key which we can use to track the versions.

Git Trees

Trees solve two problems:

• the need to remember the hashes of our objects and its version
• the possibility to storing groups of files.

The best way to think about a tree is like a folder in the file system.  To create a tree you have to follow two steps:

# Create and populate a staging area
git update-index --add --cacheinfo 100644 da95f8264a0ffe3df10e94eed6371ea83aee9a4d 1.json

# write the tree
git write-tree
d6916d3e27baa9ef2742c2ba09696f22e41011a1

This also gives you back a sha. Now we can read back that tree:

git cat-file -p d6916d3e27baa9ef2742c2ba09696f22e41011a1
100644 blob da95f8264a0ffe3df10e94eed6371ea83aee9a4d    1.json

At this point our object database looks as follows:

To modify the file, we follow the same steps:

# Add a blob
echo {"id": 1, "name": "kenneth truyers"} | git hash-object -w --stdin
42d0d209ecf70a96666f5a4c8ed97f3fd2b75dda

# Create and populate a staging area
git update-index --add --cacheinfo 100644 42d0d209ecf70a96666f5a4c8ed97f3fd2b75dda 1.json

# Write the tree
git write-tree
2c59068b29c38db26eda42def74b7142de392212

That leaves us with the following situation:

We now have two trees that represent the different states of our files. That doesn’t help much, since we still need to remember the sha-1 values of the trees to get to our content.

Git Commits

One level up, we get to commits. A commit holds 5 pieces of key information:

1. Author of the commit
2. Date it was created
3. Why it was created (message)
4. A single tree object it points to
5. One or more previous commits (for now we’ll only consider commits with only a single parent, commits with multiple parents are merge commits).

Let’s commit the above trees:

# Commit the first tree (without a parent)
echo "commit 1st version" | git commit-tree d6916d3
05c1cec5685bbb84e806886dba0de5e2f120ab2a

# Commit the second tree with the first commit as a parent
echo "Commit 2nd version" | git commit-tree 2c59068 -p 05c1cec5
9918e46dfc4241f0782265285970a7c16bf499e4

This leaves us with the following state:

Now we have built up a complete history of our file. You could open the repository with any git client and you’ll see how 1.json is being tracked correctly. To demonstrate that, this is the output of running git log:

git log --stat 9918e46
9918e46dfc4241f0782265285970a7c16bf499e4 "Commit 2nd version"
 1.json     | 1 +
 1 file changed, 1 insertions(+)
05c1cec5685bbb84e806886dba0de5e2f120ab2a "Commit 1st version"
 1.json | 1 +
 1 file changed, 1 insertion(+)

And to get the content of the file at the last commit:

git show 9918e46:1.json
{"id": 1, "name": "kenneth truyers"}

We’re still not there though, because we have to remember the hash of the last commit. Up until now, all objects we have created are part of git’s object database. One characteristic of that database is that it stores only immutable objects. Once you write a blob, a tree or a commit, you can never modify it without changing the key. You can also not delete them (at least not directly, the git gc command does delete objects that are dangling).

Git References

Yet another level up, are Git references. References are not a part of the object database, they are part of the reference database and are mutable. There are different types of references such as branches, tags and remotes. They are similar in nature with a few minor differences. For the moment, let’s just consider branches. A branch is a pointer to a commit. To create a branch we can write the hash of the commit to the file system:

echo 05c1cec5685bbb84e806886dba0de5e2f120ab2a > .git/refs/heads/master

We now have a branch master, pointing at our first commit. To move the branch, we issue the following command:

git update-ref refs/heads/master 9918e46

This leaves us with the following graph:

And finally, we’re now able to read the current state of our file:

git show master:1.json
{"id": 1, "name": "kenneth truyers"}

The above command will keep working, even if we add newer versions of our file and subsequent trees and commits as long as we move the branch pointer to the latest commit.

All of the above seems rather complex for a simple key-value store. We can however abstract these things so that client applications only have to specify the branch and a key. I’ll come back to that in a different post though. For now, I want to discuss the potential advantages and drawbacks of using git as a NoSQL database.

Data efficiency

Git is very efficient when it comes to storing data. As mentioned before, blobs with the same content are stored only once because of how the hash is calculated. You can try this out by adding a whole bunch of files with the same content into an empty git repository and then checking the size of the .git folder versus the size on disk. You’ll notice that the .git folder is quite a bit smaller.

But it doesn’t stop there, git does the same for trees. If you change a file in a sub tree, git will only create a new sub tree and just reference the other trees that weren’t affected. The following example shows a commit pointing at a hierarchy with two sub folders:

Now if I want to replace the blob 4658ea84, git will only replace those items that are changed and keep those that haven’t as a reference. After replacing the blob with a different file and committing the changes the graph looks as follows (new objects are marked in red):

As you can see, git only replaced the necessary items and referenced the already existing items.

Although git is very efficient in how it references existing data, if every small modification would result in a complete copy, we would still get a huge repository after a while. To mitigate this, there’s an automatic garbage collection process. When git gc runs, it will look at your blobs. Where it can it will remove the blobs and instead store a single copy of the base data, together with the delta for each version of the blob. This way, git can still retrieve each unique version of the blob, but doesn’t need to store the data multiple times.

Versioning

You get a fully versioned system for free. With that versioning also comes the advantage of not deleting data, ever. I’ve seen examples like this in SQL databases:

id    | name    | deleted
1     | kenneth | 1

That’s OK for a simple record like this, but that’s usually not the whole story. Data might have dependencies on other data (whether they’re foreign keys or not is an implementation detail) and when you want to restore it, chances are you can’t do it in isolation. With git, it’s simply a matter of pointing your branch to a different commit to get back to the correct state on a database level, not a record level.

Another practice I have seen is this:

id | street  | lastUpdate
1  | town rd | 20161012

This practice is even less useful: you know it was updated, but there’s no information on what was actually updated and what the previous value was. Whenever you update data, you’re actually deleting data and inserting new one. The old data is lost forever. With git, you can run git log on any file and see what changed, who changed it, when and why.

Git tooling

Git has a rich toolset which you can use to explore and manipulate your data. Most of them focus on code, but that doesn’t mean you can’t use them with other data. The following is a non-exhaustive overview of tools that I can come up with of the top of my mind.

Within the basic git commands, you can:

• Use git diff to find the exact changes between two commits / branches / tags / …
• Use git bisect to find out when something stopped working because of a change in the data
• Use git hooks to get automatic change notifications and build full-text indices, update caches, publish data, …
• Revert, branch, merge, …

And then there are external tools:

• You can use Git clients to visualize your data and explore it
• You can use pull requests, such as the ones on GitHub, to inspect data changes before they are merged
• Gitinspector: statistical analysis on git repositories

Any tool that works with git, works with your database.

NoSQL

Because it’s a key-value store, you get the usual advantages of a NoSQL store such as a schema-less database. You can store any content you want, it doesn’t even have to be JSON.

Connectivity

Git can work in a partitioned network. You can put everything on a USB stick, save data when you’re not connected to a network and then push and merge it when you get back online. It’s the same advantage we regularly use when developing code, but it could be a life saver for certain use cases.

Transactions

In the above examples, we committed every change to a file. You don’t necessarily have to do that, you can also commit various changes as a single commit. That would make it easy to roll back the changes atomically later.

Long lived transactions are also possible: you can create a branch, commit several changes to it and then merge it (or discard it).

Backups and replication

With traditional databases, there’s usually a bit of hassle to create a schedule for full backups and incremental backups. Since git already stores the entire history, there will never be a need to do full backups. Furthermore, a backup is simply executing git push. And those pushes can go anywhere, GitHub, BitBucket or a self-hosted git-server.

Replication is equally simple. By using git hooks, you can set up a trigger to run git push after every commit. Example:

git remote add replica git@replica.server.com:app.git
cat .git/hooks/post-commit
#!/bin/sh
git push replica

This is fantastic! We should all use Git as a database from now on!

Hold on! There are a few disadvantages as well:

Querying

You can query by key … and that’s about it. The only piece of good news here is that you can structure your data in folders in such a way that you can easily get content by prefix, but that’s about it. Any other query is off limits, unless you want to do a full recursive search. The only option here is to build indices specifically for querying. You can do this on a scheduled basis if staleness is of no concern or you can use git hooks to update indices as soon as a commit happens.

Concurrency

As long as we’re writing blobs there’s no issue with concurrency. The problem occurs when we start writing commits and updating branches. The following graph illustrates the problem when two processes concurrently try to create a commit:

In the above case you can see that when the second process modifies the copy of the tree with its changes, it’s actually working on an outdated tree. When it commits the tree it will lose the changes that the first process made.

The same story applies to moving branch heads. Between the time you commit and update the branch head, another commit might get in. You could potentially update the branch head to the wrong commit.

The only way to counter this is by locking any writes between reading a copy of the current tree and updating the head of the branch.

Speed

We all know git to be fast. But that’s in the context of creating branches. When it comes to commits per second it’s actually not that fast, because you’re writing to disk all the time. We don’t notice it, because usually we don’t do many commits per second when writing code (at least I don’t). After running some tests on my local machines I got into a limit of about 110 commits/second.

Brandon Keepers showed some results in a video a few years ago and he got to about 90 commits / second which seems in line of what hardware advances may have brought.

110 commits/second is enough for a lot of applications, but not for all of them. It’s also a theoretical maximum on my local development machines, with lots of resources. There are various factors that can affect the speed:

Tree sizes

In general, you should prefer to use lots of subdirectories instead of putting all documents in the same directory. This will keep the write speed as close to the maximum as possible. The reason for that is that every time you create a new commit, you have to copy the tree, make a change to it and then save the modified tree. Although you might think that affects size as well, that’s actually not the case because running git gc will make sure to save it as a delta instead of as two different trees. Let’s take a look at an example:

In the first case, we have 10.000 blobs stored in the root directory. When we add a file we copy the tree that contains 10.000 items, add one and save it. This could be a potentially lengthy operation, because of the size of the tree.

In the second case we have 4 levels of trees, with each 10 sub trees and 10 blobs at the last level (10 * 10 * 10 * 10 = 10.000 files):

In this case, if we want to add a blob, we don’t need to copy the entire hierarchy, we just need to copy the branch that leads to the blob. The following image shows the trees that had to be copied and amended:

So, by using sub folder, instead of having to copy 1 tree with 10.000 entries, we can now copy 5 trees with 10 entries, which is quite a bit faster. The more your data grows, the more you’ll want to use sub folders.

Combining values into transactions

If you need to do more than 100 commits/second, chances are you don’t need to be able to roll them back on an individual basis. In that case, instead of committing every change, you could commit several changes in one commit. You can write blobs concurrently, so you could potentially write 1000s of files concurrently to disk and then do 1 commit to save them into the repository. This has drawbacks, but if you want raw speed, this is the way to go.

The way to solve this is to add a different backend to git that doesn’t immediately flush its contents to disk, but writes to an in-memory database first and then asynchronously flushes it to disk. Implementing this is not that easy though. When I was testing this solution using libgit2sharp to connect to a repository, I tried using a Voron-backend (which is available as open-source, as well as a variant that uses ElasticSearch). That improved speed quite a bit, but you lose the benefit of being able to inspect your data with any standard git tool.

Merging

Another potentially pain point is when you are merging data from different branches. As long as there are no merge conflicts, it’s actually a rather pleasant experience, as it enables a lot of nice scenarios:

• Modify data that needs approval before it can go “live”
• Run tests on live data that you need to revert
• Work in isolation before merging data

Essentially, you get all the fun with branches you get in development, but on a different level. The problem is when there IS a merge conflict. Merging data can be rather difficult because you won’t always be able to make out how to handle these conflicts.

One potential strategy is to just store the merge conflict as is when you’re writing data and then when you read, present the user with the diff so they can choose which one is correct. Nonetheless, it can be a difficult task to manage this correctly.

Conclusion

Git can work as a NoSQL database very well in some circumstances. It has its place and time, but I think it’s particularly useful in the following cases:

• You have hierarchic data (because of its inherent hierarchical nature)
• You need to be able to work in disconnected environments
• You need an approval mechanism for your data (aka you need branching and merging)

In other cases, it’s not a good fit:

• You need extremely fast write performance
• You need complex querying (although you can solve that by indexing through commit hooks)
• You have an enormous set of data (write speed would slow down even further)

So, there you go, that’s how you can use git as a NoSQL database. Let me know your thoughts!

The post Git as a NoSql database appeared first on Kenneth Truyers.

Most developers love open source software, and often we come across a piece of software that we’re writing and think “it would be great if that already existed as an open source package”, but then, it doesn’t. Since we’re writing software for a company, the natural tendency is then to implement it in-house. The thing is, if you had that thought, chances are someone else might have had the same thought. So, why shouldn’t we open source it? From a business perspective, telling your boss you want to write open source software on company might not make much sense to him. After all, he’s paying you to write software for everyone. If you argument is going to be “open source is cool”, you probably won’t get very far.

Apart from being cool, developing open source software as a company actually has a lot of benefits.

Before I iterate over what I think the advantages are, I want to make one thing clear. When we write open source software on company time, we don’t want to open source the company’s unique selling point. It’d be foolish to think any company would allow that. When I talk about open sourcing company software, I’m talking about general purpose software. Software you need to develop to support your domain. This could be an interface to a database, some utility functions you often use or any other piece that could be usable outside of the context of your company.

Open source software advantages

Quality

Open source software is, by its nature, public. That means that any hack you implement, any security mistakes will be visible for everybody. Therefor, when you’re working on something public, it’s natural to be a lot more cautious about how you develop. But that’s only one way open source improves  quality.

Another reason quality improves is that it forces you to decouple it from your domain. In in-house software it often becomes tempting to include domain logic inside external libraries, because it’s just easier, faster. In the long run, you will have less separation of concerns which affects maintainability. Writing general purpose open source software, forces you to decouple it.

Testing

If your project is popular, a community will form around it. Once you have a community, they will start using it and possibly discover bugs before you run into them. If you have an active community they might even create a pull request to solve the bug for you. That’s free testing and bug fixing for the company.

New features / scenarios

Similar to the scenario with bug detection and public testing, it’s possible someone runs into a use case that the current software doesn’t handle. If they really need it, they might decide to implement it and create a pull request.

Exposure

If your software is of a high quality, it will start becoming popular. Being open source doesn’t mean it needs to be white-labeled. So, every time someone gets in contact with the software, your company’s logo will be there. This gives exposure of your company’s name to a potential large audience of developers.

There are plenty of examples of tech companies that are open sourcing software, to name a few:

• Uber: https://uber.github.io/
• Spotify: https://github.com/spotify
• Google: Android
• AirBnb: http://nerds.airbnb.com/open-source/
• Facebook: React

If you go through the list, you’ll see they’re not open sourcing their main selling point, but parts that orthogonal to their business model (ie: you won’t see Google open sourcing their search algorithm, or Facebook’s timeline rules). The above is also a list of companies that are hot among developers. A lot of developers want to work in these companies, precisely because of their openness. Having exposure of your company to developers could attract new talent. That’s not to say that this is the ultimate hiring strategy, but it’s another channel which could yield some interesting results.

Developer satisfaction

Good developers are, in my opinion, passionate about their work. They will feel happier if they can work on something that solves more than the daily problems they’re running into. Having a happier development team aids in productivity, retaining talent and general atmosphere in the company. Working on something which is open source and not “just” for the company might also trigger them to spend some hobby time on it which again is free work for the company.

Developer involvement

Sometimes developers move on from the company. One of the first things you do, is remove their access to the code repository. Even if they feel they did something useful and want to continue using it, they’ll have to work on it privately on a copy they have. Technically they’re not allowed to, but that’s not the reality.

On the other hand, if the code is open source, and they move on, they can still contribute to it. Likely they will use it in the next company they’re at. This will minimize the loss of knowledge in the team and again get more development time on the software for free. This is something that I experienced first hand when moving on from a company. The software we had written was by no means popular, but I found it to be useful, so I introduced it at my new company. Now we’re happily contributing to it when we need it.

It’s a win-win-win. It’s good for my new company, because they get software that was already developed, it’s good for my old company, because I (and my colleagues) are adding new features and bug fixes for their software and it’s good for me, because I didn’t have to do it all over again.

Conclusion

A lot of the above advantages are only real if your software becomes popular. But what do you have to lose? If it doesn’t become popular it’s just the same as developing it in-house which was the starting point.

Another argument, although not very tangible, is that it’s just the right thing to do. We live in a world where we can only achieve what we’re achieving by standing on the shoulders of others. I’m sure 99,99% of closed source software is using open source software somewhere, so why not be a good citizen and contribute back to the community?

The post Open source software on company time appeared first on Kenneth Truyers.

However, in the long term, I believe there’s more benefit in doing the exact opposite. Although counter intuitive, I think spreading the knowledge of the domain throughout the development team has a lot of advantages.

Knowledge sharing

If you assign a task only to a developer that has worked on a feature before, you concentrate the knowledge of that part of the domain in that developer. While having a lot of knowledge is good for a single developer, it’s not good for the team. Having the knowledge spread out over the team unlocks a bunch of advantages:

Consistency

Initially, a developer with more knowledge of the code will complete a task faster. That is a logical fact. The same is true for the opposite: if you give the task to a developer with no knowledge of the code at all, it will take him considerably more time and effort to complete it. In an ideal world, a team is always the same size and no one ever leaves, takes holidays or is off sick. In reality however, teams change, have other responsibilities and are in general not constant.

By having knowledge concentrated on a developer/feature basis, you will see a lot of highs and lows in the productivity of the team. The highs occur on moments that the team is in a stable period with no one leaving the company, no sick days and/or holidays. The lows happen when someone leaves. All of a sudden a lot of knowledge has left the company and someone needs to pick that up. The same happens when someone goes on holiday. In a way, a holiday is even worse, because the rest of the team will just sit and wait to start “that feature that touches invoicing” until John, who knows all about invoicing, is back from holiday.

If you spread the knowledge however, you will eventually get to a point where development speed is consistent. Holidays are spread over the year and the team just picks up any work that’s available. The same happens when someone leaves the company. What’s more is that a new developer can be trained by anyone, not just by that one other guy who also has the knowledge required (provided there IS actually another one around).

Responsibility

If you know that no one will look at the code you’re writing (at least not until you’re gone), there’s a natural tendency to get sloppy. Conversely, knowing that this code will be viewed tomorrow by your colleague tends to put you on edge. This is not just because developers are sloppy or because of a bad developer. To some extent, every developer is prone to this, regardless of how good or professional they are. It’s simple human nature.

Communication

Having shared knowledge moves code ownership from the individual to the team. With everyone at the same wavelength, it will improve communication within the team, and mistakes will get picked up by the team, not by individuals. Individuals can have good days and bad days. A team usually zeroes that out.

Code Reviews

If you’re doing code reviews (and you should: https://www.kenneth-truyers.net/2016/04/08/code-reviews-why-and-how/), rotating the team will also improve their quality. If you review code of which you have never seen the context, it’s a lot harder to assess the quality. Either it takes a lot of time to review, because you have to read and analyze all the surrounding bits, or, more likely, developers tend to think, “looks decent enough, I suppose that’s handled somewhere else”.

If you know the context, it’s much easier to spot bugs, suggest alternate patterns and provide valuable feedback. Without knowing context, code reviews are often reduced to formatting checks, something that’s better left to automation.

Code Quality

By having a broad knowledge of the entire domain, chosen solutions tend to fit better into the whole. It’s easy to provide a narrow solution for the problem at hand, but it’s very difficult to provide a generic solution that will be scalable in light of where the business is heading. By sharing the knowledge, you prevent tunnel vision.

Developer Satisfaction

Development is a creative activity. Nothing kills creativity more than repetition and boredom. By working on the same part over and over again, developers get bored, leave for other, more exciting places or simply go into standby mode, doing what they need to do and nothing more. By having people work on different parts, they will feel more as part of an organization, an idea and can see the goals. That creates a motivating environment and will make them want to do the best possible job.

Planning

Related to the point on consistency, planning workload also becomes a lot easier. You no longer have to check each developer’s schedule and you don’t have to cut out requested holidays (which also helps for developer satisfaction). Because there are multiple developers who can do a job, you can just assign the task to any developer that’s available.

Terms and conditions

Rotating the team around features has a lot of advantages, as explained above. Obviously, don’t take this advice to the extreme. Don’t let your DBA design your front page, don’t let your UX specialist optimize DB queries and, for the love of god, don’t let your PR spokesmen implement your log in page.
Developers all have their specialties, that’s OK, but they should be technical specialties. What you want to avoid is that developers become specialists in a thin slice of the domain.

Another thing to keep in mind is the team size. I’ve found the above guidelines to be useful in small teams. Once your team grows beyond 7-8 developers, I prefer to take the other extreme: separate the teams. The boundaries between the knowledge should be more clearly defined and any team should consider code by a different team as if it were third-party code. This allows teams to be very focused. It also means that external interfaces should be very clear, well documented and above all, stable.

Conclusion

Sometimes doing the non-intuitive thing, is more beneficial than doing what seems natural. Rotating a team around features increases consistency, code quality, responsibility and developer’s satisfaction. While going for short term wins might be tempting, the long term benefits are clear.

The post Avoiding code ownership appeared first on Kenneth Truyers.

KISS

Database migrations, contrary to popular belief, are not rocket science. Essentially, you want to execute some scripts whenever you release a new version of your software and possibly execute some scripts to undo those in case you want to rollback a faulty deployment. Building such a thing is not very difficult. It won’t have all the bells and whistles that a tool such as Redgate has, but the knowledge required is far less and, more importantly, instantly understandable for any new hires on a team. If all you need is upgrade and downgrade, you can use the code from this article with your modifications and tweaks. It’s based on some simple conventions without any possibilities for configuration or customization, but again, YAGNI (you aren’t gonna need it). If you end up needing it, then you can just modify the script and get on with more interesting stuff.

The code is very simple in its setup and it works like this:

• A table MigrationScripts is created in the database which contains all scripts that have been executed and at what date
• On startup of the application (or whichever moment you choose), it scans a folder for .sql scripts with the naming convention YYYYMMDD-HHMM-<some_name>.sql
• The code then does a diff to see which scripts have already been executed in the database
• It then runs the scripts that haven’t been executed yet, in the order of the date parsed from the naming convention

Database migrations: upgrading

The example that I’ll be showing here is something I used previously on an ASP.NET MVC app, which was the only client accessing the database. In case you have multiple applications accessing the same database, you probably want to extract this code into a separate application so you can deploy that application whenever a database update is needed. I’ll show the simple version though.

NOTE: I’m using Dapper in this example, as we where using it in said project, but this could easily be done with any other micro-ORM or ADO.NET directly.

public static void Run()
{
    string conString = ConfigurationManager.ConnectionStrings["sql_migrations"]
                                           .ConnectionString;
    using (var con = new SqlConnection(conString))
    {
        // check if the migrations table exists, otherwise execute the first script (which creates that table)
        if (con.ExecuteScalar<int>(@"SELECT count(1) FROM sys.tables
                                            WHERE T.Name = 'migrationscripts'") == 0)
        { 
            con.Execute(GetSql("20151204-1030-Init"));
            con.Execute(@"INSERT INTO MigrationScripts (Name, ExecutionDate) 
                                                VALUES (@Name, GETDATE())", 
                                                new { Name = "20151204-1030-Init" });
            }

        // Get all scripts that have been executed from the database
        var executedScripts = con.Query<string>("SELECT Name FROM MigrationScripts");

        // Get all scripts from the filesystem
        Directory.GetFiles(HostingEnvironment.MapPath("/App_Data/Scripts/"))
                 // strip out the extensions
                 .Select(Path.GetFileNameWithoutExtension)
                 // filter the ones that have already been executed
                 .Where(fileName => !executedScripts.Contains(fileName))
                 // Order by the date in the filename
                 .OrderBy(fileName => 
                    DateTime.ParseExact(fileName.Substring(0, 13), "yyyyMMdd-HHmm", null))
                 .ForEach(script =>
                 {
                     // Execute each on of the scripts
                     con.Execute(GetSql(script));
                     // record that it was executed in the migrationscripts table
                     con.Execute(@"INSERT INTO MigrationScripts (Name, ExecutionDate) 
                                                         VALUES (@Name, GETDATE())", 
                                                         new { Name = script });
                });
    }
}

static string GetSql(string fileName) =>
    File.ReadAllText(HostingEnvironment.MapPath($"/App_Data/Scripts/{fileName}.sql"));

That’s it, about 20 lines of code (comments and line breaks don’t count ;-)) for a fully working database migration infrastructure.

Database migrations: downgrading

In cases where you want to be able to roll back the database, you could add another convention: all rollback scripts have the same name but with _rollback appended to the filename. Then you can add a separate function which takes as an argument the name of the script you want to roll back. From there on, it’s a case of loading the correct rollback scripts, sorting them, executing them and removing the records from the MigrationScripts table. All in all, another 20 lines.

Conclusion

The above code allows you to:

• Store all your database changes in Git
• Do code reviews on SQL scripts (by anyone, including DB Admins)
• Pull the repo with a fresh database, run the application and get started (handy for new hires)
• Test the migrations locally and in every test environment
• Do data migrations (or any SQL you want to write for your migrations)
• Be flexible: you own the code, so anything is possible (convention change, extract it, deploy it separately, etc. )

It does have a little cost of ownership, as you may need to modify it sometimes. I’d argue however that the cost is smaller than having to know about your ORM’s migration intricacies or learn how to use a database management tool.

Also, this code is fully certified “Works on my machine”-ware. You can use it, tweak it, ask me a question about it, but don’t ask me to create a NuGet package of it, as it would go right back to the place I wanted to avoid with this code snippet.

The post Database migrations made simple appeared first on Kenneth Truyers.

EsLint is quite a nice alternative for JsHint and is very flexible. While JsHint certainly has its benefits and comes out of the box with a lot of configurable options, EsLint allows you to configure your own custom rules.

These custom EsLint rules can be added to their library and released to the community as optional extra checks, they can be company specific to enforce a certain coding style or they can be project specific.

In this post I want to talk about creating project specific custom EsLint rules. It’s easily transferrable to a more common use plugin if you want to (by publishing to NPM). I found creating a project specific plugin has a few more hurdles, so I’ll explain that.

Analyzing code

Asbtract Syntax Trees

Before we start writing custom EsLint rules, I first want to show how code is analyzed and how we can plug in to that. In order to analyze code we must first build an abstract syntax tree. In this case we want to build an ES 6 abstract syntax tree (AST). An AST is essentially a data structure which describes the code. The next example shows some sample code and the corresponding syntax tree:

var a = 1 + 1;

The above visualization can also be presented as a pure data structure, here in the form of JSON:

{
    type: "VariableDeclaration",
    declarations: [{
        type: "VariableDeclarator",
     id: {
         type: "Identifier",
            name: "a"
        },
        init: {
            type: "BinaryExpression",
            left: {
                type: "Literal",
                value: 1,
                raw: "1"
            },
            operator: "+",
            right: {
                type: "Literal",
                value: 1,
                raw: "1"
            }
        }
    }],
    kind: "var"
}

When we have this syntax tree, you could walk the structure and then write something like this for each node:

if(node.type === "VariableDeclarator" && node.id.name.length < 2){
    console.log("Variable names should be more than 1 character");
}

Writing custom EsLint rules

Since all of this AST-generation and node-walking is not specific to any rule, it can be externalized, and that’s exactly what EsLint gives us. EsLint builds the syntax tree and walks all the nodes for us. We can then define interception points for the nodes we want to intercept. Apart from that, EsLint also gives us the infrastructure to report on problems that are found. Here’s the above example rewritten as an EsLint rule:

module.exports.rules = {
    "var-length": context => ({
        VariableDeclarator: (node) => {
            if(node.id.name.length < 2){
                context.report(node, 'Variable names should be longer than 1 character');
            }
        }
    })
};

This can then be plugged in to EsLint and it will report the errors for any Javascript code you throw at it.

EsLint plugins

In order to write a custom EsLint rule, you need to create an EsLint plugin. An EsLint plugin must follow a set of conventions before it can be loaded by EsLint:

• It must be a node package (distributed through NPM, although there’s a way around it, read on …)
• Its name must start with eslint-plugin

The documentation mentions a way to write custom rules in a local directory and running them through a command-line option. This still works, but is deprecated and will soon break in newer versions of EsLint. It’s recommended to go the plugin-route as described in this post.

Creating the plugin

With the above requirements, we can go two routes:

• Use YeoMan and the corresponding EsLint generator
• Create your own package

The generator sets you up with a nice folder structure, including tests, a proper description and some documentation. However, if you just want to write some quick rules, I find it easier to just create a folder and the structure myself. Essentially, you need two files:

• package.json (remember, it has to be an NPM package)
• index.js, where your rules will live

Here’s a basic version of the package.json:

{
  "name": "eslint-plugin-my-eslist-plugin", // remember, the name has to start with eslint-plugin
  "version": "0.0.1",
  "main": "index.js",
  "devDependencies": {
    "eslint": "~2.6.0"
  },
  "engines": {
    "node": ">=0.10.0"
  }
}

And this is what index.js looks like, with our custom EsLint rule:

module.exports.rules = {
    "var-length": context => ({
        VariableDeclarator: (node) => {
            if(node.id.name.length < 2){
                context.report(node, 'Variable names should be longer than 1 character');
            }
        }
        // , more interception points (see https://github.com/estree/estree)
    })
    // more rules
};

Installing the plugin

As I mentioned before, if you want to share your plugin, you can distribute it via NPM. This doesn’t always make sense though as you might have project specific rules. In those cases, you can just create the folder with your plugin locally and commit it to your code repository. For it to work, you still need to install it as a node package though. You can do that with the following NPM command:

npm install -S ./my-eslint-plugin

This will install the package from the local folder my-eslint-plugin. That way, you can keep the rules locally to your project and still use them while running EsLint.

Configuring the plugin

For EsLint to recognize and use the plugin we have to notify it through the configuration. We need to do two things:

• Tell it to use the plugin
• Switch on the rules

To tell it to use plugin, we can add a plugins node into the configuration, specifying the name of the plugin (without the “eslint-plugin”-prefix):

"plugins": [
    "my-eslint-plugin"
]

Next we need to define the rules:

"rules": {
    "my-eslint-plugin/var-length": "warn"
}

With the plugin installed, you can now run EsLint and it will report on one letter variable names.

Example

While this is all nice, the above rule is probably not very useful, since there’s already a built-in rule for that (http://eslint.org/docs/rules/id-length).

As for general styling rules, EsLint has probably most of them already covered, and the ones it hasn’t are probably quite obscure. Custom EsLint rules come in handy on a project-level basis.

As an example, I’m currently working on an Angular 1 project. The intention is to port this over to a different framework soon. Because of that, we want to make sure we’re as independent of Angular as possible. There are certain things we can do just as easily in plain JS instead of using angular’s utility methods. For others, we can use different libraries that we can port over as well when we port the application.

Now, we don’t want to go off and change all these occurrences at once, because that would be a lot of upfront work. Ideally, we want the following:

• Get notified when there’s a call to an angular-method which could be done easily in plain JS in the module we’re working on
• Get notified on the CI-server (with a warning) if an angular-method is used
• Once we get rid of the warnings for that angular-method, fail the build on the CI-server if that call is detected again

So, as an example, here are a few rules we defined in our project:

module.exports.rules = {
    "no-angular-copy": context => ({
        MemberExpression: function(node) {
            if (node.object.name === "angular" && node.property.name === "copy") {
                context.report(node, "Don't use angular.copy, use cloneDeep from lodash instead.");
            }
        }
    }),
    "no-angular-isDefined": context => ({
        MemberExpression: function(node) {
            if (node.object.name === "angular") {
                if(node.property.name === "isDefined") {
                    context.report(node, "Don't use angular.isDefined. Use vanilla JS.");
                } else if (node.property.name === "isUndefined") {
                    context.report(node, "Don't use angular.isUndefined. Use vanilla JS");
                }
            }
        }
    })
};

We then enabled the rules with a warning in our configuration. As soon as we notice no more warnings for one of these rules, we will switch them to errors. The CI-build is configured to fail when EsLint finds an error. On top of that we have the EsLint plugin for VsCode, which looks like this in the editor:

This combination ensures that we clean up angular calls while we continue development and that no new calls will be introduced.

Sidenote: the rules here are not foolproof since someone could assign angular to a temp variable and then call the methods on the temp variable.  Be that as it may, we want to catch the general use case with a simple rule. We could probably write a more thorough analyzer, but it would take a lot of time. Since all of this code still needs to go through code reviews, we don’t worry too much about it.

Other possibilities

The above example is something that was very convenient for our use case, but the possibilities are endless. Here are a few things you could achieve with this:

• Ensure a user message is shown when HTTP calls are initiated and that they’re properly removed once it ends.
• Ensure jQuery isn’t used when we’re using a SPA-framework (or only in certain modules)
• When using jQuery, ensure you’re always calling event-handlers using the .on method instead of the shorthand ,click and similar

There are plenty of possibilities for custom EsLint rules and most of it depends on your project. What other ideas do you have?

Existing plugins

Of course, there are plenty of existing EsLint plugins for existing frameworks on NPM already. If you’re using one of these frameworks, it’s worth checking out the rules and see if you could benefit from enabling some of these

More information on writing custom EsLint rules can be found in the offical documentation

The post Writing custom EsLint rules appeared first on Kenneth Truyers.

In this week’s post I want to do the same thing but for Javascript. ES6 (ES2015) is becoming more and more mainstream, but in terms of usage I mostly see the more common arrow-functions or block-scoping (with let and const).

However, iterators and generators are also a part of Javascript and I want to go through how we can use them to create deferred execution in Javascript.

Iterators

An iterator is an object that can access one item at a time from a collection while keeping track of its current position. Javascript is a bit ‘simpler’ than c# in this aspect and just requires that you have a method called next to move to the next item to be a valid iterator.

The following is an example of function that creates an iterator from an array:

let makeIterator = function(arr){
    let currentIndex = 0;
    return {
        next(){
            return currentIndex < arr.length ? 
             {
                value: arr[currentIndex++],
                done : false
             } :
             { done: true};
        }
    };
}

We could now use this function to create an iterator and iterate over it:

let iterator = makeIterator([1,2,3,4,5]);
while(1){
    let {value, done} = iterator.next();
    if(done) break;
       console.log(value);
}

Iterables

An iterable is an object that defines its iteration behavior. The for..of loop can loop over any iterable. Built-in Javascript objects such as Array and Map are iterables and can thus be looped over by the for..of construct. But we can also create our own iterables. To do that we must define a method on the object called @@iterator or, more conveniently, use the Symbol.iterator as the method name:

let iterableUser = {
    name: 'kenneth',
    lastName: 'truyers',
    [Symbol.iterator]: function*(){
        yield this.name;
        yield this.lastName;
    }
}

// logs 'kenneth' and 'truyers'
for(let item of iterableUser){
    console.log(item);
}

Generators

Custom iterators and iterables are useful, but are complicated to build, since you need to take care of the internal state. A generator is a special function that allows you to write an algorithm that maintains its own state. They are factories for iterators. A generator function is a function marked with the * and has at least one yield-statement in it.

The following generator loops endlessly and spits out numbers:

function* generateNumbers(){
  let index = 0;
  while(true)
    yield index++;
}

A normal function would run endlessly (or until the memory is full), but similar to what I discussed in the post on yield return in C#, the yield-statement gives control back to the caller, so we can break out of the sequence earlier.

Here’s how we could use the above function:

let sequence = generateNumbers(); //no execution here, just getting a generator

for(let i=0;i<5;i++){
    console.log(sequence.next());
}

Deferred Execution

Since we have the same possibilities for yielding return values in Javascript as in C#, the only what’s missing to be able to recreate LINQ in Javascript are extension methods. Javascript doesn’t have extension methods, but we can do something similar.

What we’d like to do is to be able to write something like this:

generateNumbers().skip(3)
                 .take(5)
                 .select(n => n * 3);

It turns out, we can do this, although we need to take a few hurdles.

To attach methods to existing objects (similar to what extension methods do in c#), we can use the prototype in Javascript. Generators however all have a different prototype, so we can’t easily attach new methods to all generators. Therefore, what we need to do is make sure that they all share the same prototype. To do that, we can create a shared prototype and a helper function that assigns the shared prototype to the function:

function* Chainable() {}
function createChainable(f){
  f.prototype = Chainable.prototype;
  return f;
}

Now that we have a shared prototype, we can add methods to this prototype. I’m also going to create a helper method for this:

function createFunction(f) {
  createChainable(f);
  Chainable.prototype[f.name] = function(...args) {
    return f.call(this, ...args);
  };
  return f;
}

In the above method:

• It makes sure the function itself is also chainable, by calling createChainable
• Then it attaches the method to the shared protoype (using the name of the function). The method receives the arguments, which gets passed on to that method while supplying the correct this-context.

With this in place we can now create our “extension methods” in Javascript:

// the base generator
let test = createChainable(function*(){
      yield 1;
      yield 2;
      yield 3;
      yield 4;
      yield 5;
});

// an 'extension' method
createFunction(function* take(count){
  for(let i=0;i<count;i++){
      yield this.next().value;
  }
});

// an 'extension' method
createFunction(function* select(selector){
  for(let item of this){
      yield selector(item);
  }
});

// now we can iterate over this and this will log 2,4,6)
for(let item of test.take(3).select(n => n*2)){
    console.log(item);
}

Note that in the above method, it doesn’t matter whether we first take and then select or the other way around. Because of the deferred execution, it will only fetch 3 values and do only 3 selects.

Caveat

One problem with the above is that it doesn’t work on standard iterables such as Arrays, Sets and Maps because they don’t share the prototype. The workaround is to write a wrapper-method that wraps the iterable with a method that does use the shared prototype:

let wrap = createChainable(function*(iterable){
    for(let item of iterable){
           yield item;
     }
});

With the wrap function, we can now wrap any array, set or map and chain our previous function to it:

let myMap = new Map();
myMap.set("1", "test");
myMap.set("2", "test2");
myMap.set("3", "test3");

for(let item of wrap(myMap).select(([key,value]) => key + "--" + value)
                           .take(3)){
    console.log(item);
}

One more thing I want to add is the ability to execute a chain, so that it returns an array (for c# devs: the ToList-method). This method can be added on to the prototype:

Chainable.prototype.toArray = function(){
  let arr = [];
  for(let item of this){
      arr.push(item);
  }
  return arr;
}

Conclusion

If we implement the above, it allows us to write LINQ-style Javascript:

mySet.set("1", "test");
mySet.set("2", "test2");
mySet.set("3", "test3");

wrap(mySet).select(([key,value]) => key + "--" + value)
           .take(3)
           .toArray()
           .forEach(item => console.log(item));

Obviously, this only works in ES2015 and it’s probably not a good idea to actually write LINQ in Javascript using this method (and besides, there are already other implementations of LinqJS), but it does demonstrate the power of Iterators and Generators in Javascript.

The post Iterators and Generators in Javascript appeared first on Kenneth Truyers.

Even if most developers have heard of yield return it’s often misunderstood. Let’s start with an easy example:

IEnumerable<int> GetNumbers()
{
    yield return 1;
    yield return 2;
    yield return 3;
}

While the above has no value for anything serious, it’s a good example to debug to see how the yield return statement works. Let’s call this method:

foreach(var number in GetNumbers())
    Console.WriteLine(number);

When you debug this (using F11, Step Into), you will see how the current line of execution jumps between the foreach-loop and the yield return statements. What happens here is that each iteration of the foreach loop calls the iterator method until it reaches the yield return statement. Here the value is returned to the caller and the location in the iterator function is saved. Execution is restarted from that location the next time that the iterator function is called. This continues until there are no more yield returns.

A first use case of the yield statement is the fact that we don’t have to create an intermediate list to hold our variables, such as in the example above. There are a few more implications though.

Yield return versus traditional loops

Let’s have a look at a different example. We’ll start with a traditional loop which returns a list:

IEnumerable<int> GenerateWithoutYield()
{
    var i = 0;
    var list = new List<int>();
    while (i<5)
        list.Add(++i);
    return list;
}

foreach(var number in GenerateWithoutYield()) 
    Console.WriteLine(number);

These are the steps that are executed:

1. GenerateWithoutYield is called.
2. The entire method gets executed and the list is constructed.
3. The foreach-construct loops over all the values in the list.
4. The net result is that we get numbers 1 to 5 printed in the console.

Now, let’s look at an example with the yield return statement:

IEnumerable<int> GenerateWithYield()
{
    var i = 0;
    while (i<5)
        yield return ++i;
}

foreach(var number in GenerateWithYield())
    Console.WriteLine(number);

At first sight, we might think that this is a function which returns a list of 5 numbers. However, because of the yield-statement, this is actually something completely different. This method doesn’t in fact return a list at all. What it does is it creates a state-machine with a promise to return 5 numbers. That’s a whole different thing than a list of 5 numbers. While often the result is the same, there are certain subtleties you need to be aware of.

This is what happens when we execute this code:

1. GenerateWithYield is called.
2. This returns an IEnumerable<int>. Remember that it’s not returning a list, but a promise to return a sequence of numbers when asked for it (more concretely, it exposes an iterator to allow us to act on that promise).
3. Each iteration of the foreach loop calls the iterator method. When the yield return statement is reached the value is returned, and the current location in code is retained. Execution is restarted from that location the next time that the iterator function is called.
4. The end result is that you get the numbers 1 to 5 printed in the console.

Example: infinite loops

Now you might think that since both example behave exactly the same, that there’s no difference in which one we use. Let’s modify the example a bit to show where the differences lie. I’m going to make two small changes:

• Instead of looping in the generator until we reach 5, I’m going to loop endlessly:

  IEnumerable<int> GenerateWithYield()
  {
      var i = 0;
      while (true)
          yield return ++i;
  }
  
  IEnumerable<int> GenerateWithoutYield()
  {
      var i = 0;
      var list = new List<int>();
      while (true)
          list.Add(++i);
      return list;
  }
  

• Instead of iterating directly over the list, I’m going to take 5 items of the list:

  foreach(var number in GenerateWithoutYield().Take(5))
      Console.WriteLine(number);
  
  foreach(var number in GenerateWithYield().Take(5))
      Console.WriteLine(number);

When we do this, the difference is clear. Following the previously described steps, in the case of the method without yield, the loop will never finish as it will keep looping forever inside the GenerateWithoutYield-method when it’s called in the first step (until it throws an OutOfMemoryException). In the case of the GenerateWithYield-method however, we get a different behavior. Because the Take-method is actually implemented with a yield return operator as well, this will succeed. The method only gets called until the Take-method is satisfied.

Example: multiple iterations

Another side effect of the yield return statement is that multiple invocations will result in multiple iterations. Let’s have a look at an example:

IEnumerable<Invoice> GetInvoices()
{
    for(var i = 1;i<11;i++)
        yield return new Invoice {Amount = i * 10};
}

void DoubleAmounts(IEnumerable<Invoice> invoices)
{
    foreach(var invoice in invoices)
        invoice.Amount = invoice.Amount * 2;
}

var invoices = GetInvoices();
DoubleAmounts(invoices);

Console.WriteLine(invoices.First().Amount);

Read through the above code sample and try to predict what will be written to the console.

What do you think the output is here? 20? In fact, the result is 10. Let’s see why:

• When the line var invoices = GetInvoices(); is executed we’re not getting a list of invoices, we’re getting a state-machine that can create invoices.
• That state machine is then passed to the DoubleAmounts-method.
• Inside the DoubleAmounts-method we use the state-machine to generate the invoices and we double the amount of each of those invoices.
• All the invoices that were created are discarded though, as there are no references to them.
• When we return to the main method, we still have a reference to the state-machine. By calling the First-method we again ask it to generate invoices (only one in this case). The state-machine again creates an invoice. This is a new invoice and as a result, the amount will be 10.

Because this is non-obvious behavior, tools such as Resharper will warn you about multiple iterations.

Real life usage

It’s pretty neat that we can write seemingly infinite loops and get away with it, but what can we use it for in real life? In broad terms, I’ve found two main use cases (all other use cases I found are a subclass of these two).

Custom iteration

Let’s say we have a list of numbers. We now want to display all the numbers larger than a specific number. In a traditional implementation that might look like this:

IEnumerable<int> GetNumbersGreaterThan3(List<int> numbers)
{
    var theNumbers = new List<int>();
    foreach(var nr in numbers)
    {
        if(nr > 3)
            theNumbers.Add(nr);
    }
    return theNumbers;
}
foreach(var nr in GetNumbersGreaterThan3(new List<int> {1,2,3,4,5})
    Console.WriteLine(nr);

While this will work, it has a disadvantage: we had to create an intermediate list to hold the items. The flow can be visualized as follows:

You can see in the above image, how the first list is created, then iterated and filtered into a new list. This new list is then iterated again.

We can avoid this intermediate list by using yield return:

IEnumerable<int> GetNumbersGreaterThan3(List<int> numbers)
{
    foreach(var nr in numbers)
    {
        if(nr > 3)
            yield return nr;
    }
}
foreach(var nr in GetNumbersGreaterThan3(new List<int> {1,2,3,4,5})
    Console.WriteLine(nr);

Now, the execution looks very different:

In this diagram it’s clear that we only iterate the list once. When we get to the items that are needed, control is ceded to the caller (the foreach-loop in this case)

Stateful iteration

Since the method containing the yield return statement will be paused and resumed where the yield-statement takes place, it still maintains its state. Let’s take a look at the following example:

IEnumerable<int> Totals(List<int> numbers)
{
    var total = 0;
    foreach(var number in numbers)
    {
        total += number;
        yield return total;
    }
}

foreach(var total in Totals(new List<int> {1,2,3,4,5})
    Console.WriteLine(total);

The above code will output the values 1,3,6,10,15. Because of the pause/resume behavior, the variable total will hold its value between iterations. This can be handy to do stateful calculations.

Deferred execution

All of the above samples have one thing in common: they only get executed as and when necessary. It’s the mechanism of pause/resume in the methods that makes this possible. By using deferred execution we can make some methods simpler, some faster and some even possible where they were impossible before (remember the infinite number generator).

The entire LINQ part of C# is built around deferred execution. Let’s see a few sample how deferred execution can make things more efficient:

var dollarPrices = FetchProducts().Take(10)
                                  .Select(p => p.CalculatePrice())
                                  .OrderBy(price => price)
                                  .Take(5)
                                  .Select(price => ConvertTo$(price));

Suppose we have 1000 products. If the above method did not have deferred execution, it would mean we would:

• Fetch all 1000 products
• Calculate the price of all 1000 products
• Order 1000 prices
• Convert all the prices to dollars
• Take the top 5 of those prices

Because of deferred execution however, this can be reduced to:

• Fetch 10 products
• Calculate the price of 10 products
• Order 10 prices
• Convert 5 of these prices to dollars

While maybe a contrived example, it shows clearly how deferred execution can greatly increase efficiency.

Side note: I want to make clear that deferred execution in itself does not make your code faster. Inherently, it has no effect on the speed or efficiency of your code. The value of deferred execution is that it allows you to optimize your code in a clean, readable and maintainable way. This is an important distinction.

Conclusion

The yield-keyword is often misunderstood. Its behavior can seem a bit strange at first sight. However, it’s often the key to creating efficient code that is maintainable at the same time. Its main use cases are custom and stateful iteration which allow you to create simple yet powerful code. The yield-keyword is what’s powering the deferred execution used in LINQ and allows us to use it in our code. I hope this article helped explaining the semantics of the yield-keyword and the effects and implications it has on calling code. Feel free to ask any questions in the comments!

The post Yield return in C# appeared first on Kenneth Truyers.

For me, it’s an honor to be part of this community. Not only does it feel good to be recognized for the work I’ve been doing but it’s an even greater opportunity to learn from people and build a professional network. The Open MVP day is exactly about that.

It was a busy week, as I just started a new contract, flew out to London and then had to rush to Rome to attend the Open MVP day. Lots of travel, lots of attention required and all in a short time span. While I was happy to be able to do all of that, I can’t say I wasn’t relieved to be back at home as well.

People I’ve met

Apart from the technical information I got, the most important part for me was to get to know as many people as possible and build a network of peers. It’d be great to continue some of the conversations I’ve had with people there and build a relationship with mutual benefits. On the night I arrived I quickly caught up with the Spanish delegation (mostly with Dachi Gogotchuri, Sergio Guerra and Asier Villanueva) and we had a good chat over a beer about what we like (and don’t like) about the technologies we work in. It was particularly interesting to see what grievances are shared and which ones are probably just my own problem :-). It was really great to see so many people with the same shared interests and a commitment to keep learning and exploring technology. If you’re like me (a geek), you probably recognize the feeling where you have a lot of ideas and thoughts and no one to share them with (at least not in person).

Apart from fellow MVP’s, we also had the chance to meet some of the technical evangelists from Microsoft. They’re experts in building and maintaining communities. It was very interesting to hear some new ideas from Alejandro Campos on how to build a community and wake interest in technology. I can’t wait to start and put these ideas in action to foster the local community and build a network of like-minded people here in Mallorca.

Things I’ve learned

Apart from networking, there are obviously technical sessions. While I found that most sessions where rather introductory sessions, I do want the highlight the session by Ricardo Peres on ElasticSearch. While also an introductory session, this one was particularly interesting as I just started a project with heavy usage of ElasticSearch. I definitely learnt a lot in that session and hope to soon start to apply that knowledge in real life.

The warning signs

If there’s one thing I was a bit weary about, I’d say it’s the effect of the echo chamber. This has nothing to do with the organizers or the sessions, but with the very nature of a vendor specific event. Since all attendees are Microsoft MVP’s, the focus naturally lies on Microsoft technology. Even though I’m mostly Microsoft oriented, I like to venture into related technologies to compare, contrast and learn from them. This doesn’t mean I find MS tech worse (or better) than other tech stacks, it just means that, while attending an event that focuses on a particular vendor, it’s important to not get soaked up by it and keep an open mind.

On the other hand, I also like to mention that there a lot of MVP’s that didn’t only specialize in Microsoft tech. As an example, Nicola Iarocci, is a python specialist on the server, but works with MS tech on the client side. His perspective was particularly interesting as it shows that it isn’t necessary to be only “devoted” to MS tech to become an MVP. It shows that the movement towards openness from Microsoft is not just hollow words.

Conclusion

All in all, I’d say that my first Open MVP day was a great success and I can’t wait to attend my first MVP summit later this year. As I’ve been told, it’s a fantastic opportunity to meet peers from all over the world as well as get an insight on “how the sausage is made” at Microsoft.

I want to thank the organizers and everybody I’ve met for making this a great first experience and hope to see everyone at the next gathering.

The post Impressions as a rookie Microsoft MVP appeared first on Kenneth Truyers.

The namespace pattern described in my other post has a few drawbacks:

• It relies on a single global variable to be the application’s global. That means there’s no way to use two versions of the same application or library. Since they both need the same global name, they would overwrite each other.
• The syntax can become a bit heavy if you have deeply nested namespaces (eg: myapplication.services.data.dataservice)

In this post I want to show a different pattern: a javascript sandbox. This pattern provides an environment for modules to interact without affecting any outside code.

Sandbox constructor

In the namespace pattern, there was one global object. In the javascript sandbox this single global is a constructor. The idea is that you create objects using this constructor to which you pass the code that lives in the isolated sandbox:

new Sandbox(function(box){
    // your code here
});

The object box, which is supplied to the function will have all the external functionality you need.

Adding Modules

In the above snippet, we saw that the sandboxed code receives an object box. This object will provide the dependencies we need. Let’s see how this works. The Sandbox constructor is also an object, so we can add static properties to it. In the sample below we’re adding a static object modules. This object contains key-value pairs where the key indicates the module name and the value is a function which returns the module.

Sandbox.modules = {
    dom: function(){
        return {
            getElement: function(){},
            getStyle: function(){}
        };
    },
    ajax: function(){
        return {
            post = function(){},
            get = function(){}
        };
    }
};

With this in place, let’s now look at how we pass the modules to the sandboxed code. For that, we’ll have a look at a first version of the Sandbox constructor:

function Sandbox(callback){    
    var modules = [];
    for(var i in Sandbox.modules){
        modules.push(i);
    }
    for(var i = i < modules.length;i++){
        this[modules[i]] = Sandbox.modules[modules[i]]();
    }
    callback(this);
}

First we iterate over all the modules and push the names of all of them into an array. Next, we get each module from the static modules object and assign it to the current instance of the box. Lastly we pass the instance to the sandboxed code. That ensures the box has access to those modules.

Improvements

While the above is a good proof of concept, it requires some modifications to make it more versatile and safer to use.

Enforce constructor usage

First of all, let’s make sure that it’s always called as a constructor and not just with a regular function-call:

function Sandbox(callback){
    if(!(this instanceOf Sandbox){
        return new Sandbox(callback);
    }
}

Allow module specification

Next, we want to be able to define which modules we are going to use so that only those modules will be initialized and passed. We do this by accepting an array of module names and then only adding those modules to the box, instead of iterating over all the modules. That makes our constructor a bit simpler:

function Sandbox(modules, callback){    
    if(! (this instanceOf Sandbox){
        return new Sandbox(modules, callback);
    }

    for(var i = i < modules.length;i++){
        this[modules[i]] = Sandbox.modules[modules[i]]();
    }
    callback(this);
}

Optional arguments

We want to make the modules argument optional. If it’s not provided, we will use all the modules. We also want to add the ability to pass in the modules one by one as strings, instead of in an array. For that we need to do a bit of argument parsing and again iterate over all the modules:

function Sandbox(){
    // transform arguments into an array
    var args = Array.prototype.slice.call(arguments); 
    // the last argument is the callback
    var callback = args.pop(); 
    // modules is either an array or individual parameters
    var modules = (args[0] && typeof args[0] === "string" ? args : args[0];

    if(!modules){
        modules = [];
        for(var i in Sandbox.modules){
            modules.push[i];
        }
    }
}

Common instance properties

Since we’re passing in the instance of the box to the sandboxed code, we can add some predefined properties to each instance so that all sandboxed code has access to these:

function Sandbox(){
    // ...

    this.sandboxVersion = "1.0.1";
    callback(this);
}

Arguments destructuring

Currently, the client-code has to access the modules through the box-instance. It would be nicer if we could pass in the modules as separate arguments. This makes the dependencies even more explicit. To do so, instead of calling the callback directly, we can use apply to execute the callback. Also, instead of initializing the modules as properties on the sandbox, we save them in an array:

function Sandbox(){
    // ...
    var moduleInstances = modules.map(function(m){
        return Sandbox.modules[m]();
    });

    callback.apply(this, moduleInstances);
}

The complete Javascript sandbox

When we put everything together, our constructor looks like this:

function Sandbox(){
    // parse the arguments    
    var args = Array.prototype.slice.call(arguments),
    callback = args.pop(),
    modules = (args[0] && typeof args[0] === "string") ? args : args[0];

    // add properties for all sandboxes
    this.applicationVersion = "1.0.2";

    // ensure constructor call
    if (!(this instanceOf Sandbox)){
        return new Sandbox(modules, callback);
    }

    // add all modules if no modules were passed
    if(!modules){
        modules = [];
        for(var i in Sandbox.modules){
            modules.push(i);
        }
    }

    // initialize and add all modules to the sandbox
    var moduleInstances = modules.map(function(m){ 
        return Sandbox.modules[m](); 
    }); 

    // execute the code
    callback.apply(this, moduleInstances);
}

Sandbox.modules = {
    dom: function(){
        return {
            getElement: function(){},
            getStyle: function(){}
        };
    },
    ajax: function(){
        return {
            get: function(){},
            post: function(){}
        };
    }
}

With the sandbox in place, here are a few example usages:

new Sandbox('dom', function(dom){
    console.log(this.sandboxVersion);
    var element = dom.getElement();
});

new Sandbox(function(dom, ajax){
    console.log(this.sandboxVersion);
    var element = dom.getElement();
    ajax.post();
});

new Sandbox(['ajax', 'dom'], function(ajax, dom){...});

Conclusion

The Javascript sandbox let’s you isolate code from outside factors. It also allows you to explicitly define dependencies which reduces coupling and makes it easier to test. While there are other patterns to do this, and certain framework have this built in, this could be a good pattern to use if you’re still working with ES5 and no frameworks.

The post Javascript sandbox pattern appeared first on Kenneth Truyers.

What is technical debt?

When you start writing code you usually have a choice: either do it quick and messy or do it, as we developers tend to call it, “the right way”. The quick way is obviously better for the business as it delivers value earlier.

From a business point of view, no one really cares in what state the code is in. If it works, the business is happy. So then, considering that a healthy code base takes more time and is thus more expensive, why should the business have to pay more for a healthy code base, when it doesn’t really concern them?

Although a bad or untested code base may deliver business value, if it’s left uncontrolled, it’s not necessarily good in the long run. An unhealthy code base is hard to maintain and will develop stability issues. This will affect future efforts to add more business value.

In lay man’s terms, technical debt is the amount of mess left behind in a code base from quick fixes.

Benefits of technical debt

From the previous description, it might seem obvious that technical debt is a bad thing and should be avoided at all costs. That’s not entirely true though. Sometimes the cost of accruing technical debt is less than the cost of having to release later. Debt in the financial world has obvious advantages, and if managed correctly it can be a tool to improve yourself.

Taking a mortgage on a house when you have studied how to pay it back, can be a good investment. Waiting until you have saved the entire amount would be impossible for most people and thus you have a cost of opportunity. On the other hand, using a credit card like it’s a free for all pass and buying anything that you desire is probably not a good way of managing your financial situation.

In software development, the same can be said. When you need to push out that release before Christmas, it’s possibly a good idea to implement something quickly so it’s there when the opportunity is big. On the other hand, pushing out all features as soon as possible without regards to code quality, architecture and tests will sooner or later result in a slower process and reduced quality of the application.

Managing technical debt

As with financial debt, technical debt can beneficial. And also in line with how we think about financial debt, the key to balancing cost and advantages is to make sure you have a good strategy for controlling debt. Here are a few ways I found have helped me in keeping technical debt under control.

Default to avoiding technical debt

The default way of writing code in your organization should be to write well-factored, flexible code with decent test coverage. Acquiring debt is not a decision that should be taken lightly, therefore, when it’s taken it should be taken deliberately and not by coincidence because someone didn’t feel like writing good code today.

Communicate the consequences

There’s a typical conversation between a developer and a manager. If you are a developer, I’m sure you have heard it as well. It goes something like this:

Manager: I need feature X. How much time do you think it will take you?
Developer: 1 week
Manager: Hmm, it needs to go online in two days though, as big event X is in two days
Developer: OK, I’ll see what I can do
Manager: OK
Developer: OK

Familiar? I thought so. This happens all the time and the problem here is not bad standards or bad employees. The problem is communication. Here is the same conversation, but with the thoughts of both in brackets:

Manager: I need feature X. How much time do you think it will take you?
Developer: 1 week (1 day of thinking, 2 days coding and testing, 1 refactoring, 1 days extra testing)
Manager: Hmm, it needs to go online in two days though, as big event X is in two days
Developer: OK, I’ll see what I can do (I’ll cut down on the thinking and testing)
Manager: OK (I’m a great manager, I just managed to get something done in 2 days which normally takes a week)
Developer: OK (ugh, always the same, we just can’t write decent code here)

It’s important to realize that various factors are at play here:

• The manager wants feature X in two days, not because he’s a tyrant, but for a good reason: it earns more money if implemented earlier.
• The manager walks away with idea that it can be done in two days. If this happens often, he will realize that pushing developers works, after all, in two days he’s going to see that it works indeed. He doesn’t know what happens in the code base, so it’s normal that the next time around he’ll try to negotiate the estimate.
• The developer wants to satisfy the need of the business, but feels incapable of doing so in both short and long term

As a developer we have an obligation to communicate better to the business. Here’s my improved version of this conversation:

Manager: I need feature X. How much time do you think it will take you?
Developer: 1 week
Manager: Hmm, it needs to go online in two days though, as big event X is in two days
Developer: It’s impossible to do this feature well in 2 days, it needs a week to be done properly.
Manager: OK, but we don’t have a week. If it takes a week, there’s no point as we won’t earn as much money from it.
Developer: What I can do is take a shortcut and do it quickly. That would require me to rearrange some things and I need to go back later to fix it.
Manager: OK, that sounds reasonable (great, I’m going to get it done in time)
Developer: OK (great, I will make sure the feature gets implemented soon and then I’ll need to go back to make sure nothing gets left behind that can cause trouble in the future)

In this conversation, a middle ground is found. The feature will be implemented, and the technical debt is accounted for and managed. The consequences are well understood by the business and can be dealt with accordingly.

Track

Just as with financial debt, you want to know what debt you have and how long it would take you to get rid of it (even if you’re never going to get rid of all of it). In the above story, it would be wise to create a task for cleaning up the code base and writing tests after the feature was implemented. This way, it’s visible to everyone that there was technical debt acquired.

The way you track technical debt depends on your process. In an agile process, we have created a new type of story before. Apart from user stories, tasks and bugs we’d have another story type called “technical debt”. This allows us to see in a quick view how much technical debt we have and whether it’s becoming a problem.

Apart from tracking technical debt as and when you create it, it’s sometimes also necessary to track technical debt that you spot. This could be either legacy code or it could be some big refactoring that you and the team feel is necessary for the code base to be flexible towards future developments.

Depending on what situation you’re in, you could incorporate a certain percentage of the time to working on technical debt stories. make sure the business is aware of this and approves.

Repay your debt

Communication and tracking don’t serve any purpose if you don’t repay your debt. You have to make sure that technical debt stories are dealt with on a regular basis. When deciding which stories to tackle you need to factor in a few properties:

• Age: Just as with financial debt, technical debt comes with an interest. The longer you leave code in a bad state, the bigger the impact it will have: it’ll create more bugs and people will forget why and how something was implemented (remember, it’s bad code, so it’s probably obscure by nature)
• Impact: There’s a difference between a class that has some formatting issues and an entire subsystem that doesn’t have any tests. Tackle those issues that have the biggest impact first. (to continue the analogy: pay of the credit card with 20% interest rate before you pay of the one with 2% interest rate)

Apart from the need to repay your debt, there are also different ways you can choose to repay it:

• Repay it completely: replace the code or refactor it to a good solution
• Partially repay it: Instead of implementing a good solution, implement a different solution that has less interest
• Don’t repay it at all: just deal with the “interest”. This can be a good option if the cost is minimal, the code is hardly ever changed and replacing it would be very costly

Dealing with legacy code

Considering that we default to writing good code and all technical debt is communicated and dealt with appropriately, a normal project should never have so much technical debt that it impacts the business. However, there are situations where we don’t have these values from the beginning of the project: legacy projects.

Dealing with legacy projects is a totally different ball game. Often it’s difficult to identify the good code (or worse, there is no good code). Furthermore, it’s difficult to identify which parts of the code are causing most problems and how to solve them (aka: it’s difficult to estimate the interest).

To deal with this situation it’s best to create a metaphorical fork in the road, from which point you default to writing good code. All legacy code should be isolated as much as possible. Once legacy code is isolated, you can then decide that all modifications to that legacy should leave the code in a better state. A good book to read on this topic is Working Effectively with Legacy Code by Michael Feathers.

Conclusion

Technical debt is inevitable in software projects. Instead of trying to avoid it, we should try to manage it as effectively as possible. When managed correctly, technical debt can be a powerful tool to help your business grow faster without impacting the long term goals.

The post Technical debt: managing code quality appeared first on Kenneth Truyers.

Of all the practices implemented to improve code quality, such as unit testing, continuous integration, continuous deployment, daily stand-ups, I find the most important one is doing proper code reviews.

Code reviews have a lot of advantages:

• It’s much easier to spot problems with other people’s code than with your own
• Knowledge of the codebase is spread through the team, thereby avoiding the pitfall where certain developers have sole knowledge of a subsystem
• General knowledge of the team is improved as new methods or practices are visible to everyone
• Adherence to code standards are enforced

It can be a bit awkward in the beginning, but once you have found a good flow, you’ll notice a big change in how software is developed, not only in terms of quality but also in terms of the morale and coherence of the development team.

Rules for reviewing code

For code reviews to work in your benefit, a certain set of rules need to be followed. It’s easy to just start using some software, but if you aren’t reviewing code consistently, it won’t bring you all the benefits listed above. Here are a few the rules I have experienced to positively influence the gains of code reviews as well as satisfaction among developers.

Small commits, good commit messages

Having a lot of small grained commits with decent commit messages makes it much easier to review code as everything will be explained step-by-step. It also makes it easier for the reviewer to see the thinking behind the implementation.

As an added benefit, git blame will serve as your living documentation for all your code.

Review the code, not the developer

Since you’re reviewing someone’s work, it might be tempting to critique that person. However, if this culture persists, you’ll more often than not get developers that aren’t happy anymore or who will start hiding their not-so-nice code.

This doesn’t mean you should let code pass that doesn’t live up to the standard, it just means you should critique the code instead of the person. This can be a subtle difference. Instead of saying “You didn’t follow the standard here”, which can sound accusatory (intentional or not), say “this code should be formatted differently”. Note the difference in tone.

This responsibility also falls on the developer. As a developer you have to get into the mindset that you are not your code. Even if a comment might sound accusatory, don’t take it personally and remember that it’s a comment on the code, not on you or your behavior.

Multiple reviewers

Always assign more than 1 person to review the code. That way, if there are any disagreements, it will be easily solved by a majority vote. If you assign only 1 developer, you might run into a discussion without end between the developer and the reviewer.

Have a checklist

Having a list of things to look out for, makes it easier to conduct code reviews systematically. Here’s my personal list of things to check for (in order of priority):

• Function: does it do what it needs to do? (A good spec goes a long way here)
• Bugs
• Test coverage
• Adherence to architecture and patterns
• Duplicate code
• Code style and formatting

Having a checklist also makes it easier for the developer to run through it before submitting the code for review.

When to review

This depends a bit on the confidence of the team and how well they work together.

In teams with low to medium confidence, I would opt for a feature-branch strategy where code is reviewed first and only then  integrated into the main line. In this case, you have to make sure that code is reviewed as soon as possible, since you don’t want any branches to live for a long time only to see that your properly reviewed code can’t be integrated anymore without merge conflicts.

In teams with a high confidence level, I would opt to integrate directly into the main line and do reviews after the fact. The reason this can only be done in high confidence teams is multiple:

• It might open you up to code reviews that are never completed
• A developer can ignore the code review
• Bad code can be committed

Review process

Whether you sit down together in front of the screen or have software to be able to complete your task, make sure reviewing code is as accessible as possible. The last thing you want is that developers come to see it as a chore. A fast code review process will yield more code reviews and better results.. I had very good experiences with Kiln. It does more than just code reviews, but I particularly liked their interface (YMMV).

Quick tip: Review only tests
If you have limited time and have good test coverage (or it’s enforced by your build process), you can choose to only review the tests. The reasoning is that if the tests are good, the implementation will be good as well. By reviewing the tests you will see what the public API looks like, so you’ll get a good feel about the code. Use this tip sparingly though, a full review is still useful.

Pitfalls

Apart from following the above rules, there are a few pitfalls and anti-patterns to look out for.

Gaming the system

If you don’t have complete buy-in from the whole team, this issue might come up. It happens when 2 or more devs on the team decide to approve each other’s code without properly reviewing it. They’re technically following the rules, but in practice the code is not reviewed at all. A possible remedy is to make sure that the rule of 2 or more reviewers is appended with a rule that says you have to select two different people all the time.

Review gate

This problem can manifest itself in a few ways:

• One dev never approves of any review so code is unnecessarily blocked in the review. If this happens frequently, find out what the underlying issue is.
• All code needs to pass past one developer before it hits main. While this can sometimes be good for a short while, it’s never a good plan to do this long-term. If really necessary, than at least that role should be switched regularly. Otherwise developers might stop caring about their code at all.
• Complete lockdown: this happens where developers are taken away permission to touch the main line at all. If you really have big code quality issues it can be a good temporary measure, but otherwise this is the fastest way to sink your team’s morale.

Conclusion

Code reviews are, in my experience, the most valuable tool for improving software quality, distributing knowledge and enforcing a common coding standard.

If you’re just starting out with code review, keep in mind that it needs perfecting. Don’t worry when it’s not yet creating that huge change you expected. If you improve the process bit by bit and get more experienced, you’ll soon see the benefits.

The post Code Reviews: why and how? appeared first on Kenneth Truyers.

Build 2016 is finished and as always it was great to see Microsoft bringing new opportunities to businesses and developers. Unfortunately I wasn’t able to attend, but luckily, the live stream of all the important sessions, especially for the keynotes, made up for that. These are the announcements that excited me the most.

Microsoft Bot Framework

Completely unexpected but a very cool way of building new applications and solutions. The idea behind the bot framework is to use conversations as an application framework. The big challenge for developers is to make the interaction with bots as natural as possible. To that end, Microsoft is offering a framework plus a set of intelligence services such as speech and text recognition and a wide variety of other cognitive services. This should enable developers to build clever bots that can automate the things we do on websites at the moment. I’m not sure it will replace websites anytime soon as Microsoft claims, but it definitely has some benefits over traditional applications. User interface design becomes kind of obsolete, and if you think about it, we have been using language to communicate our intentions forever, so if we can properly crack the key to that, we could see very interesting applications.

Obviously Microsoft wouldn’t be Microsoft if they didn’t connect their existing services to this new framework. Skype and Cortana will be tied in and soon you’ll see new integrations pop up in these tools.

Bash on Windows

A few years ago, the good old April’s fool day joke would be Microsoft releasing a Linux distro or working together with Linux and open source in general. The news is the same, only this time it’s for real. Through an integration with native Ubuntu binaries, windows developers can now use long known bash tools such as grep, awk, sed, … This definitely opens up a lot of possibilities, not in the least for making it easier to follow online tutorials.

HoloLens

Already announced earlier this year, but now it’s for real: the HoloLens dev-kit is now going out to developers. We’ve seen some impressive demos from Microsoft so far, but now it will be interesting to see what the rest of the world can do with it. This is the first real test for HoloLens. If it really is that impressive as the demos we saw from Microsoft, we’re up for some mind blowing applications in the next couple of months. Furthermore, because of a few design changes on the actual headset, users who tried it out reported a better field of view, which was one of the points of critique up until now.

Azure is getting bigger

Azure already consists of a huge set of services that make the life of developers easier. At Build 2016, Microsoft added a bunch of new services to grow Azure even more. These were all announced:

• Azure IoT Starter Kits are now available for purchase from partners
• Azure IoT Hub device management and Gateway SDK will be available later in Q2
• A new service, Azure Functions is now in preview
• DocumentDb supports a MongoDb protocol now
• Azure Developer Tools
• Microsoft Cognitive Services is in preview

Xamarin

Probably the most awaited announcement. As everyone was hoping, Xamarin will now come bundled with Visual Studio. That’s great news for developers that were using the paid version before as it was quite expensive. Not only does it come with paid versions of Visual Studio but also with the free community edition. To top it off, they also announced open sourcing the Xamarin core SDK. These announcements were certainly above expectations. While everyone was hoping for the Visual Studio bundling, no one dared to hope for inclusion in the free product and even less on having it available as open source.

Desktop App Converter

While one of the big disappointments of the last months was the discontinuation of the Android porting project, project Astoria, Microsoft now did release another porting tool, this time to convert Win32 application to UWP. Any app based on Win32 and .NET can be converted to the AppX format. Furthermore, work is still continuing on project Islandwood, the porting tool for iOS apps. Let’s hope these converters can make a dent in the app gap.

What are you planning to do with these new services?

The post Build 2016 announcements appeared first on Kenneth Truyers.

First of all, what happened? I’m not going to go too deep on what actually happened (there’s plenty of information about that), but essentially, because of a copyright dispute a package was removed and replaced with a different one. Because of this, a whole slew of popular packages that depended on this broke. All projects depending on those consequently broke as well, resulting in broken builds all over the world.

To add insult to injury, the package that caused most issues was leftPad, a seemingly trivial module that should be easy for any developer to implement themselves.

The incident has caused a lot of reflection on what the best way is to do dependency management. The discussion basically broke down into two camps:

In the one camp you have people saying that developers should not include dependencies for trivial things or on modules that consist of a one-liner. They argue that:

• Functions or one liners are too small to include as dependencies
• There’s no guarantee that what someone has written is correct
• Developers who are not capable of writing a trivial method like the one mentioned, should not be writing code at all
• Modules this trivial and this widely used, should be part of the base framework (nodejs in this case)

The other camp sees the above problem as an issue with the registry (NPM) and that there’s no problem in having small packages. They argue that:

• We shouldn’t reinvent the wheel for common functionality
• Small modules allow for more modular design and composability of applications

All good points, and I certainly agree with every argument made from both camps.

Granular or coarse composition?

I have experienced a similar problem in one of my previous projects. We were using internally distributed modules, so we didn’t have any of the aforementioned issues with registries and authorship.

The project I’m talking about was built in .NET and we were using NuGet packages. Regardless, the problem space is still the same: package and dependency management.

We were torn between two different styles of dependency management:

• Either create a single package with multiple utilities, the typical <companyname>.Utils package so to speak.
• Or create a single package for every single utility: <companyname>.LeftPad (for example)

Our problem was related to the authoring and the management of these packages:

In the first option, it was easy to publish new versions of the package. On the other hand, if you wanted only one part of it, you had to import everything and the kitchen sink.

The second option was easier from a client point of view as you could just import what you needed, but it made authoring and maintaining all these different packages quite difficult.

Dependency management and visibility

Another problem, which is true for both solutions, but magnified by the second approach, was debugging these modules. If you would encounter a problem in one of the packages, it was impossible to step into them to see what they were doing. This shouldn’t be a problem in stable packages, but that’s not always the case. If you have a single package, you can publish the symbol files (in .NET) and use them for debugging. For multiple modules, this is also possible but it creates more overhead (at authoring time, but also in start up time of your application and loading symbols).

Side note: We experienced first hand that SymbolSource, the repository for .NET symbols, is also not the most stable solution for storing symbols.

While this is a problem specific to .NET, I do think that hiding away dependencies in a node_modules folder is kind of the same, you lose visibility.

The main issue I have with traditional package management solutions (NPM, NuGet, …) is that you lose visibility on what’s happening with your code base. I don’t like the idea of having to depend on external developers and less when there’s no guarantee that these packages will always be in the same hands (a change breaks the trust you have in a package).

Not only is visibility a problem, but when there’s an actual bug in a dependency, you have to hope that the author will fix it or accept your pull request if you decide to fix it yourself. If you can’t do this, you might get stuck.

Our solution

We came up with an intermediate solution (based on a post by Nik Molnar) for package management: Instead of depending on a compiled package that is stored in the packages folder (equivalent to a package that lives in node_modules), we decided to publish source-only packages. What does this mean?

Say we have a package LeftPad. Instead of creating a package that distributes a DLL with that function, we would distribute a package that creates a class LeftPad.cs in your solution. This solves a few problems:

• You don’t need to reinvent the wheel, you can reuse existing modules that are used by many others (and thus potentially vetted)
• The code is available inside your project, so it’s visible and modifiable at all times
• If an update comes along, it will overwrite the class. Source control will show you exactly what was modified and you have great visibility over what could potentially be a breaking change
• You can make modifications to it very easily, again, any updates will highlight where your local modifications would be overwritten and it’s quite easy to manage

Another key point is that these packages are marked as development dependencies, which means they won’t be installed in subsequent levels of the dependency chain. This makes the dependency chain a lot flatter and more manageable.

A potential disadvantage is that you have to take ownership of the external code. I don’t see that as a problem though, but rather an advantage. It improves visibility and in the end you’re the one responsible for your application.

This approach is not new and has been used in previous projects:

• NETFx: https://netfx.codeplex.com/
• Quarks: https://github.com/shaynevanasperen/Quarks (this is the project that grew out of our project)

In my opinion this approach could work as well for NPM. Package authors could have an option of outputting certain files to the application directory instead of the nodes_modules folder so they can increase the visibility,

What do you think about this approach? We haven’t experienced any downsides with it (apart from some naming collisions, that were easily solved). Sound off in the comments!

The post Dependency management: strategies and pitfalls appeared first on Kenneth Truyers.

In ASP.NET MVC, applications are divided into horizontal layers, which is reflected in the project structure:

• Controllers
• Views
• Models
• Scripts
• Style

It’s a good idea to divide you application into logical parts. While the idea of horizontal slices might look like a good idea, in practice I have noticed that it’s not necessarily the only way and more often than not, not the best way. Why not?

The following is a diagram of how a standard MVC application is structured:

From the diagram above, you can see that our application has 5 layers (the standard MVC ones) and then probably a few other ones, depending on the application. Cutting through these layers are the features: all of them have styles, scripts, views, controllers and models.

There are a few reasons why this separation does not make sense:

• Changes usually happen vertically, not horizontally. As an example, if you need to add a field to your database, you need to add it to your model, do some validation in the controller, display it in the view, style it in your CSS and do something funky with it in JavaScript.
• When you make horizontal slices, you automatically limit your application to the same model over all the slices. It’s weird to have one controller that accesses the database directly and another that first goes through another layer. Because you have layered the application, everything is suddenly layered. In a vertically structured application you can choose which paradigm to use for each part of the application. Even though it doesn’t physically limit you from mixing it up, it certainly pushes you in that direction (and I have seen those exact guidelines on several projects).
• From a purely practical point of view, it’s very difficult having all these files that relate to the same feature in different folders.
• It’s an arbitrary structure. From the diagram above, you can see that the natural structure our application wants to have is vertical (all our features are vertical), but the actual structure is horizontal.

A better way to structure an application would be in vertical layers. We want all code belonging to a feature to sit together in a vertical slice (also called a feature slice) so it’s easy to work on that specific feature. That allows us to work in a very tight scope and makes our solution explorer work for us, instead of having to swap between folders all the time. A more optimal structure would look like this:

In the above diagram, you can see that our slices are following the natural structure of the application. Furthermore, if we go one level deeper, we see that not all slices have controllers, not all of them depend on a domain (why do you need a domain model for content rendering?). That means we can tailor our code to exactly what is needed for that part of the application. A folder structure could look like this:

Features
    -> Users
        - users.css
        - users.js
        - UserController.cs
        - Index.cshtml
        - Detail.cshtml
    -> Search
        - search.css
        - search.js
        - SearchController.cs
        - Index.cshtml
    -> Content
        - about-us.html
        - content.css
    -> Invoicing
        - invoices.css
        - invoices.js
        - InvoiceController.cs
        - InvoiceViewModel.cs
        - Invoice.cs
Styles
    - layout.css
Scripts
    - app.js

In the above folder structure you can see that:

• Working on a feature can be done by just opening one folder and modifying only the files in that folder, so you have less context switching going on
• Some features may have more or less horizontal layers
• Even scripts and CSS live in the same folder as the server-side code

I have used this structure in several projects and even though the code complexity is the same (you still need the same amount and type of code), I found that the perceived complexity is dramatically lower. Since the scope seems smaller, it feels like the application is just a bunch of smaller apps working together (under a shared structure, but still).

How?

Even though ASP.NET MVC comes standard in a horizontal flavor, it’s actually relatively simple to change this.

Controllers

If you use attribute routing, controllers are very easy, you can just move them wherever you want and they will still work. Yay, that was easy.

Models

Another easy one, just move them around at will, change a couple of namespaces and you’re done.

Views

Views are a bit trickier. By convention MVC looks for them under Views –> ControllerName or under Views –> Shared. To change this, we need to swap out the ViewEngine. That may sound difficult, but it’s actually rather simple. For the folder structure above, you can create the following class:

public class FeatureViewLocationRazorViewEngine : RazorViewEngine
{
    public FeatureViewLocationRazorViewEngine()
    {
        var featureFolderViewLocationFormats = new[]
        {
            "~/Features/{1}/{0}.cshtml",
            "~/Features/Shared/Views/{0}.cshtml",
        };

        ViewLocationFormats = featureFolderViewLocationFormats;
        MasterLocationFormats = featureFolderViewLocationFormats;
        PartialViewLocationFormats = featureFolderViewLocationFormats;
    }
}

Once you have this class, you can substitute the standard ViewEngine with this one in your global.asax:

ViewEngines.Engines.Clear();
ViewEngines.Engines.Add(new FeatureViewLocationRazorViewEngine());

Scripts

For scripts, we will have a shared part, which will still live in the default location (think about common modules for the entire application) and then there will be scripts that are specific to each feature. Those will live in the feature folders. Using ES6 modules makes this easy since you can use a bundler to bundle and minify everything. Because of the way modules work, it will automatically bundle all the scripts that are referenced from your entry point. Here’s an example of how something like that might work:

In the /scripts folder I have a file called app.js, which is my main point of entry. I trigger general modules from here and then based on the URL, I also trigger specific JS-files:

// Need to go 1 level up and into the feature folders
import ContactForm from '../Features/Contact/contact.js';
import Search from '../Features/Search/search.js';
import Users from './../Features/Users/logon.js';
import Invoices from './../Features/Invoices/invoices.js';

// this is a shared module that lives in the same directory
import Analytics from './analytics.js';

// This module always gets called, regardless of the page
new Analytics();

// Based on the path we activate a different module
var path = document.location.pathname.toLowerCase();
if(path.indexOf('contact') !== -1){
    new ContactForm();   
}
if(path.indexOf('search') !== -1){
    new Search();   
}
if(path.indexOf('invoices') !== -1){
    new Invoices();   
}
if(path.indexOf('logon') !== -1){
    new Users();   
}

Once you have the main entry point in place, you can use Gulp to bundle and minify all of this into a single file:

gulp.task('js', function () {
    return gulp.src('scripts/app.js')
             .pipe(jspm({ selfExecutingBundle: true }))
             .pipe(rename('app.min.js'))
             .pipe(gulp.dest('scripts'));
});

I’m using JSPM here, but it would work with any other bundler.

Styles

For our style sheets, I’m also going to use Gulp. In this example, I will be using SCSS because it allows me to include other files. First of all, I’m going to define my main entry point, let’s call it style.scss, which will live under the /styles folder. In this file, I only include references to other files:

// These are shared files that live under the same directory
@import "base/*";
@import "layout/*";

// using a glob I also include all .scss file in the feature-folders
@import "../Features/**/*.scss";

Note that I have used globs to include all files that are in the feature folders. This is not natively supported by most SCSS-compilers, but there’s a plugin (sassGlob) for Gulp that makes this possible (I’m sure there are plugins for Grunt as well, if you prefer to use Grunt). The following Gulp-task will make sure that all SCSS-files are combined, compiled to CSS and minified into one file:

gulp.task('sass', function () {
    return gulp.src('styles/style.scss')
               .pipe(sassGlob()) // Need to use a gulp plugin to make sure globs work
               .pipe(sass()))
               .pipe(rename({ suffix: '.min' }))
               .pipe(gulp.dest('styles'));
});

Conclusion

I find feature slices to be a huge improvement over the standard structure of an MVC-application. With a bit of infrastructure code this is easy to set up and it makes working with large application a lot easier.

The post Vertical slices in ASP.NET MVC appeared first on Kenneth Truyers.

In this post I’ll show how you can create an in-memory web server by using a NancyFX module so you can simulate the API’s responses and you can test all the details of the connection.

Set up

To set this up we need two components:

1. The host that will serve our fake web app
2. The fake web application

To be clear: Only the test project uses NancyFX, it’s not necessary to use NancyFX as your web framework.

1. The host

In your test project you will need the package Nancy.Hosting.Self . This package allows you to run a web server in-memory. The following code shows how to create a host (and how to dispose of it):

public class Host : IDisposable
{
    NancyHost _host;
    public Host()
    {
        _host = new NancyHost(new HostConfiguration { RewriteLocalhost = false }, 
                              new Uri("http://localhost:50001"));
        _host.Start();
    }

    public void Dispose()
    {
        _host.Stop();
        _host = null;
    }
}

Important to note here is the RewriteLocalhost option in the HostConfiguration. This value determines whether localhost-url’s are rewritten to http://+:port/ style URL’s to allow for listening on all ports. If you do this, you either need a namespace registration or admin access. Since we want our tests to be independent to the environment we need to disable it. The default value is true, so we explicitly set it to false. For more information, check the NancyFX documentation

We can use this class now in our unit tests. Since it’s a rather expensive operation (we want our unit tests to by lightning fast, see key qualities of a good unit test), I tend to set it up only once per test run. The following shows how to set it up once for a test run in xUnit. For this particular case (shared context between multiple test classes), we need to use collection fixtures:

[CollectionDefinition("Host")]
public class HostCollection: ICollectionFixture<Host> { }

This class is a marker class that will allows us to group classes in a collection. It will construct the host before the first test in the collection is executed and will dispose of it after the last test in the collection finishes. Once we have this marker class, we can declare the tests that rely on the host as follows:

[Collection("Host")]
public class RestClientTests
{
    …
}

Different testing frameworks have different methods of setting up shared contexts, so this can be different depending on the framework you’re using.

2. The fake web application

Now that we have an in-memory server, we can start building our fake we application that will respond to our web requests. A Nancy web app is a fairly simple module:

public class FakeApp : NancyModule
{
    public FakeApp()
    {
        Get["/products/{id}"] = _ =>
            Response.AsJson(new Product());
    }
}

You can have multiple classes inside your test project that inherit from NancyModule. You apply the route to any of the HTTP methods in the constructor to emulate the REST API and then return the values the API would normally return.

Tying it together

Now that we have an in-memory web server and a fake web app, we can use it to test our REST client.

Suppose we have the following REST client (our system under test):

public class RestClient
{
    IRestClient _client;
    public RestClient(string url)
    {
        _client = new RestClient(url);
    }

    public Product GetProductById(int id)
    {
        return _client.Get<Product>(new RestRequest($"/products/{id}")).Data;
    }
}

In this example, I’m using RestSharp, but this would work with WebClient, HttpClient or any other library.

To test this class, we can write the following test:

[Collection("Host")]
public class RestClientTests
{ 
    [Fact]
    public void When_getting_a_product_it_correctly_deserializes_it()
    {
         new RestClient("http://localhost:50001").GetProductById(1).ShouldNotBeNull();
    }
}

The Collection-attribute ensures that our host is up and our fake app is running. By passing in the local URL, we make sure that we target the correct URL so it targets our fake web application.

Improving our fake app

To be able to run some more interesting tests, we need to make sure that our fake web app emulates the REST API as much as possible. We don’t want to spend too much time replicating the real API so I’ve come up with an implementation that allows us to match URL’s with predefined JSON responses. To do this we will store canned JSON responses as files in the assembly and then use a convention to match routes to those responses.

To add a JSON-file to your assembly, add a new file with the .json-extension and set the Build Action to Embedded Resource in the properties.

Now we’ll match routes with these predefined responses:

Instead of defining various routes for all the API’s operations, we will now define only one. Based on the route that this method was called with we will retrieve the JSON from the assembly’s resources and then return that result:

// Use a greedy route
Get[@"/{route*}"] = _ =>
{
    // 1. Read the url and replace all dashes with dots
    var resourcePath = _.route.ToString().Replace("/", ".");

    // 2. Combine the filename with the path of the assembly and the directory
    var resourceName = $"<assemblyname>.<directory>.{resourcePath}.json";
                
    // 3. Read the content of the json file
    Response response;
    using (var stream = Assembly.GetExecutingAssembly()
                                .GetManifestResourceStream(resourceName))
        using (var reader = new StreamReader(stream))
            response = reader.ReadToEnd();
                
     // 4. Set the content type as JSON and return the response
     response.ContentType = "application/json";
     return response;
};

This method consists of 4 parts:

• First we get the route that the method was called with. We replace all slashes with dots (see the next step for why this is important).
• Then we translate this path into a path to a resource. To read a resource you have to specify the path to it in the format displayed above. If you have directories they should be separated with dots instead of slashes. This allows us to match routes with a directory structure.
• Next we read the content of the file and (implicitly) cast it to a NancyFX Response.
• Lastly, we set the content-type and return it.

With this in place, we don’t have to modify the server anymore. We can now add responses based on routes. As an example we can simulate the responses to the following routes, by creating the directory structure below:

• /products/{id}
• /deals/{id}
• /deals/{id}/products

  - deals
      - 1994
          -> products.json    // matches /deals/1994/products
      -> 1994.json            // matches /deals/1994
      -> 2441.json            // matches /deals/2441
  - products
      -> 4245.json            // matches /products/4245
  

  The above method only works for GET-requests, but you could devise a similar strategy for other HTTP methods as well.

  Conclusion

  With a bit of infrastructure code, which is reusable across projects, we can set up a quick way of testing our REST clients.

  In this post I showed a basic infrastructure set up and some improvements to make working with the fake application easier. There are still more improvements to make, such as a mechanism to match POST requests with canned responses, setting response headers and status codes and the ability to look into our test server to see what was sent to it.

The post Testing REST clients appeared first on Kenneth Truyers.

C# 7 Non-nullable reference types

What?

Nullable value types where introduced in C# 2.0. Essentially they’re just syntactic sugar around the Nullable<T> class. Non-nullable reference types are the reverse of that feature. It let’s you declare a reference type that is guaranteed not to be null.

Why?

The null reference has been called “The billion dollar mistake” (by the inventor: Tony Hoare). NullReference exceptions are all too common. The problem is two-fold: either you don’t check for them and then you might get runtime exceptions or you do check for them and then your code becomes verbose or littered with statements that have little to do with what you’re actually trying to achieve. The ability to declare a reference type as non-nullable overcomes these problems.

How?

NOTE: The syntax here is still in flux and will probably change. There are various proposals floating around and it’s still unclear what the definitive form will be. Also, where I mention “error”, it’s still unclear whether it will be a compilation error or just a warning.

First of all, the ideal syntax would be to default to non-nullable reference types. This would provide symmetry between reference and value types:

int a;     //non-nullable value type
int? b;    //nullable value type
string c;  //non-nullable reference type
string? d; //nullable reference type

However, there are millions of lines of C# out there that would break if non-nullable types would become the default, so unfortunately it has to be designed differently to keep everything backwards compatible. The currently proposed syntax is as follows:

int a;     //non-nullable value type
int? b;    //nullable value type
string! c; //non-nullable reference type
string d;  //nullable reference type

Using nullable and non-nullable types will then affect the compiler:

MyClass a;  // Nullable reference type
MyClass! b; // Non-nullable reference type

a = null;   // OK, this is nullable
b = null;   // Error, b is non-nullable
b = a;      // Error, n might be null, s can't be null

WriteLine(b.ToString()); // OK, can't be null
WriteLine(a.ToString()); // Warning! Could be null!

if (a != null) { WriteLine(a.ToString); } // OK, you checked
WriteLine(a!.Length); // Ok, if you say so

Using this syntax is OK, but it would become problematic for generic types:

// The Dictionary is non-nullable but string, List and MyClass aren't
Dictionary<string, List<MyClass>>! myDict;   

// Proper way to declare all types as non-nullable
Dictionary<string!, List<MyClass!>!>! myDict;

The above is a bit difficult to read (and type) so a shortcut has also been proposed:

// Typing ! in front of the type arguments makes all types non-nullable
Dictionary!<string, List<MyClass>> myDict;

C# 7 Local Functions

What?

The ability to declare methods and types in block scope.

Why?

This is already (kind of) possible by using the Func and Action types with anonymous methods. However, they lack a few features:

• Generics
• ref and out parameters
• params

Local functions would have the same capabilities as normal methods but would only be scoped to the block they were declared in.

How?

public int Calculate(int someInput)
{
    int Factorial(int i)
    {
        if (i <= 1)
            return 1;
        return i * Factorial(i - 1);
    }
    var input = someInput + ... // Other calcs

    return Factorial(input);
}

C# 7 Immutable Types

What?

An immutable object is an object whose state cannot be modified after its creation.

Why?

Immutable objects are offer a few benefits:

• Inherently thread-safe
• Makes it easier to use and reason about code
• Easier to parallelize your code
• Reference to immutable objects can be cached, as they won’t change

Currently it’s already possible to declare immutable objects:

public class Point
{
    public Point(int x, int y)
    {
        x = x;
        Y = y;
    }

    public int X { get; }
    public int Y { get; }
}

While the above is definitely an immutable object, the problem is that the intent is not clearly visible. One day, someone might add a setter and consumers of this type, expecting immutability, could experience different results.

How?

NOTE: Again, the syntax here is still in flux. The initial proposal suggests adding an immutable keyword:

public immutable class Point
{
    public Point(int x, int y)
    {
        x = x;
        Y = y;
    }

    public int X { get; }
    public int Y { get; }
}

When you have immutable types, a nice addition is language support for creating new instances based on a different instance:

var a = new Point(2, 5);
var b = a with { X = 1};

Conclusion

As I said before, it’s still early days, so the above syntax can (and probably will) change, but these features are very exciting and will make C# even more enjoyable to work with. I encourage everyone to have a look on GitHub to follow the current discussions around these features:

Work List of Features

The full details of the above proposals can also be found here:

Non-nullable reference types
Local functions
Immutable types

The post New features in C# 7, part 2 appeared first on Kenneth Truyers.

C# 7 Tuples

Update 22/07/2016: Tuples are planned to be a part of C# 7

What?

Tuples are a temporary grouping of values. You could compare a Tuple to a POCO-class, but instead of defining it as a class you can define it on the fly. The following is an example of such a class:

class PropertyBag
{
    public int Id {get; set;}
    public string Name {get; set;}
}
var myObj = new PropertyBag { Id = 1, Name = "test};

In the above example it wasn’t really necessary to name the concept we’re working with as it is probably a temporary structure that doesn’t need naming. Tuples are a way of temporarily creating such structures on the fly without the need to create classes

Why?

The most common reason for having a group of values temporarily grouped are multiple return values from a method. Currently, there are a few ways of doing that in C#:

Out parameters

public void GetLatLng(string address, out double lat, out double lng) { ... }

double lat, lng;
GetLatLng(myValues, out lat, out lng);
Console.WriteLine($"Lat: {lat}, Long: {lng}"); 

Using out-parameters has several disadvantages:

• It cannot be used for async-methods
• You have to declare the parameters upfront (and you can’t use var, you have to include the type)

Tuple-type

There currently already is a Tuple-type in C# that behaves like a native tuple. You could rewrite the previous method as follows:

public Tuple<int, int> GetLatLng(string address) { ... }

var latLng = GetLatLng("some address");
Console.WriteLine($"Lat: {latLng.Item1}, Long: {latLng.Item2}"); 

This does not have the disadvantages of out-parameters, but the resulting code is rather obscure with the resulting tuple having property names like Item1 and Item2.

Class / struct

You could also declare a new type and use that as the return type:

struct LatLng{ public double Lat; public double Lng;}
public LatLng GetLatLng(string address) { ... }

var ll= GetLatLng("some address");
Console.WriteLine($"Lat: {ll.Lat}, Long: {ll.Lng}"); 

This has none of the disadvantages of out-parameters and the tuple type, but it’s rather verbose and it is meaningless overhead.

How?

There are a few different use cases for tuples that will be available with C# 7 Tuples:

Tuple return types

You can specify multiple return types for a function, in much the same syntax as you do for specifying multiple input types (method arguments)

public (double lat, double lng) GetLatLng(string address) { ... }

var ll = GetLatLng("some address"); 
Console.WriteLine($"Lat: {ll.lat}, Long: {ll.lng}");

Inline tuples

You could also create tuples inline:

var ll = new (double lat, double lng) { lat = 0, lng = 0 };

Tuple deconstruction

Because the bundling of the values is not important as a concept, it’s quite possible that you don’t want to access the bundle at all, but get straight to the internal values. Instead of accessing the tuple properties as in the example of Tuple Return Types, you can also destructure the tuple immediately:

(var lat, var lng) = GetLatLng("some address");
Console.WriteLine($"Lat: {lat}, Long: {lng}");

C# 7 Record types

Update 22/07/2016: Records are probably not coming in C# 7, but will have to wait until the next version (supposedly c# 8)

What?

A record type is a simple bag of properties, a data type with only properties

Why?

Often classes or structs are merely a collection of properties. They still need a full declaration which is quite verbose. The following class demonstrates that a class with 3 properties requires quite a bit of text to declare:

class MyPoint
{
    int _x;
    int _y;
    int _z;
    MyPoint(int x, int y, int z){
        this._x = x;
        this._y = y;
        this._z = z;
    }
    public int X {get{ return this._x;}}
    public int Y {get{ return this._y;}}
    public int Z {get{ return this._z;}}
}

How?

With record types you could write the above in a single line:

class Point(int X, int Y, int Z);

You will get a few more things out of this:

• The class will automatically implement IEquatable<Point>, which means you can compare two record types based on their values instead of reference.
• The ToString-method will output the values in the record

C# 7 Pattern Matching

Update 22/07/2016: Pattern matching is planned to be partially supported in C# 7. In C# 7 only switching on types will be available. Full support for pattern matching will come in the next version (supposedly c# 8)

What?

With record types in play, we can now get pattern matching built-in. Pattern matching means that you can switch on the type of data you have to execute one or the other statement.

Why?

Although pattern matching looks a lot like if/else, it has certain advantages:

• You can do pattern matching on any data type, even your own, whereas if/else you always need primitives to match
• Pattern matching can extract values from your expression

How?

The following is an example of pattern matching:

class Geometry();
class Triangle(int Width, int Height, int Base) : Geometry;
class Rectangle(int Width, int Height) : Geometry;
class Square(int width) : Geometry;

Geometry g = new Square(5);
switch (g)
{
    case Triangle(int Width, int Height, int Base):
        WriteLine($"{Width} {Height} {Base}");
        break;
    case Rectangle(int Width, int Height):
        WriteLine($"{Width} {Height}");
        break;
    case Square(int Width):
        WriteLine($"{Width}");
        break;
    default:
        WriteLine("<other>");
        break;
}

In the sample above you can see how we match on the data type and immediately destructure it into its components.

C# 7 Non-nullable reference types

Update 22/07/2016: Non-nullable reference types are probably not coming in C# 7, but will have to wait until the next version (supposedly c# 8)

What?

Nullable value types where introduced in C# 2.0. Essentially they’re just syntactic sugar around the Nullable<T> class. Non-nullable reference types are the reverse of that feature. It let’s you declare a reference type that is guaranteed not to be null.

Why?

The null reference has been called “The billion dollar mistake” (by the inventor: Tony Hoare). NullReference exceptions are all too common. The problem is two-fold: either you don’t check for them and then you might get runtime exceptions or you do check for them and then your code becomes verbose or littered with statements that have little to do with what you’re actually trying to achieve. The ability to declare a reference type as non-nullable overcomes these problems.

How?

NOTE: The syntax here is still in flux and will probably change. There are various proposals floating around and it’s still unclear what the definitive form will be. Also, where I mention “error”, it’s still unclear whether it will be a compilation error or just a warning.

First of all, the ideal syntax would be to default to non-nullable reference types. This would provide symmetry between reference and value types:

int a;     //non-nullable value type
int? b;    //nullable value type
string c;  //non-nullable reference type
string? d; //nullable reference type

However, there are millions of lines of C# out there that would break if non-nullable types would become the default, so unfortunately it has to be designed differently to keep everything backwards compatible. The currently proposed syntax is as follows:

int a;     //non-nullable value type
int? b;    //nullable value type
string! c; //non-nullable reference type
string d;  //nullable reference type

Using nullable and non-nullable types will then affect the compiler:

MyClass a;  // Nullable reference type
MyClass! b; // Non-nullable reference type

a = null;   // OK, this is nullable
b = null;   // Error, b is non-nullable
b = a;      // Error, n might be null, s can't be null

WriteLine(b.ToString()); // OK, can't be null
WriteLine(a.ToString()); // Warning! Could be null!

if (a != null) { WriteLine(a.ToString); } // OK, you checked
WriteLine(a!.Length); // Ok, if you say so

Using this syntax is OK, but it would become problematic for generic types:

// The Dictionary is non-nullable but string, List and MyClass aren't

Dictionary<string, List<MyClass>>! myDict; // Proper way to declare all types as non-nullable Dictionary<string!, List<MyClass!>!>! myDict;

The above is a bit difficult to read (and type) so a shortcut has also been proposed:

// Typing ! in front of the type arguments makes all types non-nullable
Dictionary!<string, List<MyClass>> myDict;

C# 7 Local Functions

Update 22/07/2016: Local functions are planned to be a part of C# 7

What?

The ability to declare methods and types in block scope.

Why?

This is already (kind of) possible by using the Func and Action types with anonymous methods. However, they lack a few features:

• Generics
• ref and out parameters
• params

Local functions would have the same capabilities as normal methods but would only be scoped to the block they were declared in.

How?

public int Calculate(int someInput)
{
    int Factorial(int i)
    {
        if (i <= 1)
            return 1;
        return i * Factorial(i - 1);
    }
    var input = someInput + ... // Other calcs

    return Factorial(input);
}

C# 7 Immutable Types

Update 22/07/2016: Immutable types are currently not on the planning for C# 7, nor the next version

What?

An immutable object is an object whose state cannot be modified after its creation.

Why?

Immutable objects are offer a few benefits:

• Inherently thread-safe
• Makes it easier to use and reason about code
• Easier to parallelize your code
• References to immutable objects can be cached, as they won’t change

Currently it’s already possible to declare immutable objects:

public class Point
{
    public Point(int x, int y)
    {
        x = x;
        Y = y;
    }

    public int X { get; }
    public int Y { get; }
}

While the above is definitely an immutable object, the problem is that the intent is not clearly visible. One day, someone might add a setter and consumers of this type, expecting immutability, could experience different results.

How?

NOTE: Again, the syntax here is still in flux but the initial proposal suggests adding an immutable keyword:

public immutable class Point
{
    public Point(int x, int y)
    {
        x = x;
        Y = y;
    }

    public int X { get; }
    public int Y { get; }
}

When you have immutable types, a nice addition is language support for creating new instances based on a different instance:

var a = new Point(2, 5);
var b = a with { X = 1};

Conclusion

For more information about these and other new proposed features, head over to GitHub for the full list:

Work List of Features

NOTE: It’s still early days, the syntax of all of these can (and probably will) change. Some features might not even make it into C# 7 so we’ll have to wait until C# 8 or later. I encourage everyone to take a look on the GitHub page to examine and learn about these features. The discussion is quite lively there. If you have another good proposal, check out the same forums and maybe your feature gets the cut. It’s really nice that Microsoft has opened up all these channels, so we better make use of them!

Update 22/07/2016 If you want to try out these features, you can now download the preview version of Visual Studio “15” Preview 3 (which is different from Visual Studio 2015).

The post C# 7: New Features appeared first on Kenneth Truyers.

What is the zone?

The zone is usually described as a state-of-mind where one feels productive and hyper-focused. These are certainly good traits to have when you are programming. However, the zone also comes with tunnel-vision and a sense of being infallible. These are not good traits to have.

Being in the zone is also often described as a pleasurable feeling.

Furthermore, the zone is also related to left-side hemisphere vs right-side hemisphere of the brain thinking. (note that these are not referring to the actual location of where brain activity is measured, but just placeholders to indicate a certain type of brain activity). Generally speaking, the left side of the brain tends to control many aspects of language and logic, while the right side tends to handle spatial information, visual comprehension and creativity. With that in mind, do we really want to be in a state-of–mind where logic is less prominent and we have a tendency to be more creative? While creativity is a good trait to have when programming, it depends a lot on what type of creativity.

Negative effects

I have found personally that being in the zone is counter-productive. When I’m in the zone, I do feel very productive, write more code and find complex solutions. When I look at the same code afterwards though, I often find that it’s overly complex. My ‘creative’ solutions are not always as clear as they should be. Because of the tunnel vision you get when programming in the zone, you can loose the big picture. Code that I write in the zone comes out faster, but I need to go back to it more often. Even if it doesn’t need any changes, it’s harder to read and in the end causes more loss of time than something that took a bit longer to write but is more readable and understandable.

One of my favorite quotes about programming is by Brian W. Kernighan:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it

This certainly applies to code written in the zone. When I’m in the zone I’m more focused, so the code I write is the most clever code I can write. By the above definition then, I will not be smart enough to debug it unless I’m in the zone again, and thus, you’re abilities as a developer become dependant of being in the zone. I don’t think that’s a good idea.

Interruptions

The role of software developers has changed significantly over the past 10 or 20 years. While 20 years ago, the stereotype of a developer’s was a geek in a dark room in front of a bright screen, cranking out code, today, developers are part of an organization, a team and a development process, where interaction with customers and colleagues is often as important, if not more important, as cranking out code.

Something developers often complain about is interruptions. It’s true that an interruption gets you out of the zone, and being pulled out of focused state can really be annoying. But a developer’s job is not just about cranking out code, it’s about supporting your business, coding is just a part of that. If someone needs help, should they really wait until you ‘feel like answering them’? It’s part of the job, so we should help people out when needed. After all, we expect that treatment from other colleagues, don’t we?

Obviously there’s a limit to the amount of interruptions you can handle without losing productivity, but I don’t think an interruption should get anyone upset.

Using it correctly

As with most ‘tools’ we have available, saying that a tool is great or bad at everything is usually short-sighted. While I’m certainly opposed to thinking that being in the zone all the time is optimal, I also think it can be used effectively, given the correct circumstances.

When you’re learning a new tool or paradigm, it can be very beneficial to be in the zone, as you will be hyper-focused and can absorb much more information in a shorter time span. Coding kata’s are a perfect example where being in the zone can be very beneficial. A kata is a very focused exercise where you don’t need to keep the big picture in mind (there is none) and it often requires some creativity to solve the puzzle. These puzzles are excellent to train your mind.

Essentially, the zone CAN be used for a very tightly scoped and particularly hard problem, but I avoid it as a general state to develop software.

Conclusion

While many will certainly disagree with me (that’s perfectly fine), I would like to invite you to take a step back and think about whether you see being in the zone beneficial because it improves your productivity or whether you just enjoy the feeling of being in the zone.

I tend to steer away from being in the zone. When I feel myself getting into that state, I usually get up for a few minutes and do something else. I only allow this to happen whenever I want to practice some skills, because then there’s no downside of being in the zone. In the end, I think awareness of what the zone is, realizing when you’re in the zone and the ability to use it effectively is a valuable skill. What do you think? Sound of in the comments!

The post Programming in the zone appeared first on Kenneth Truyers.

• Exploratory tests: Performed manually by a tester
• System tests: Executed by a program or script that automates the UI (also known as Acceptance Tests or UI tests)
• Integration tests: Executed against a layer just beneath the UI (sometimes referred to as subcutaneous tests)
• Component tests: Executed against a single component of the application
• Unit tests: Test a single “unit” in the software (sometimes a class, sometimes a function)

In this post I’ll be talking about automated testing, so the exploratory tests are not part of this discussion. Because of their similar nature, I will also be grouping the integration tests and component tests in one category. The theory for the testing pyramid says that you should have a good coverage (almost complete) for your unit tests, a decent coverage for the integration tests and a small coverage for system tests.

 

 

This is widely accepted as the way to implement automated testing. While the general idea is sound, I think it shouldn’t be applied blindly. When you ask yourself the question, “how should we organize or testing efforts?”, the answer, as often in software, is “it depends”. For any given software product, there are various factors in play that could skew the pyramid.

Before we look at what influences our pyramid, we have to look at the characteristics of these tests. As you go towards the top of the top pyramid, your tests will instill more confidence, as they are end-to-end tests. Conversely the more to the bottom you go the less confidence the tests will bring you, since they only test a small part in isolation. On the other hand, at the top of the pyramid, you will have slower, more brittle and harder to write and maintain tests. A unit test tends to be more deterministic, faster and easier to write.

In an ideal world, we would like our tests to be end-to-end, fast, deterministic and easy to write and maintain. Unfortunately, that’s not possible (yet). However, instead of looking at the pyramid and focusing our test efforts on following this theory, we should take this ideal goal and strive towards a test suite that matches that description.

Tooling

When this theory was presented,we didn’t have all the tools we currently have (or they where not as easily accessible as they are now). A few of the characteristics have been skewed as a result of this. To give a few examples:

• Cloud infrastructure (VM’s, containers) provided us with cheap “hardware” on demand. This could solve part of the problem of slow UI tests. Instead of trying to write fast UI tests, we can just throw hardware at it. (that doesn’t mean you should write slow tests though, but sometimes it’s more cost effective to add hardware instead of manpower)
• Services like BrowserStack and SauceLabs became available, allowing you to spin up tests on a variety of platforms, without a big investment.
• Testing frameworks have improved. BDD has become quite popular and as a result a lot of the frameworks have been adding features.
• Application frameworks have been adapted to be more loosely coupled and more flexible in the way we run them. As an example, any ASP.NET application can now be self-hosted, through the use of OWIN. If you combine that with an in-memory database, it can speed up integration tests.

Knowing that the tools have improved, we can already see that our pyramid, should perhaps be a bit taller, with a little less focus on the bottom part, and some more focus on the top (confidence!). This is still on a global level though. Depending on the application you can still change the pyramid to the need of the application.

Application

The nature of the application plays a big role in where you should put your emphasis in testing. Each layer of the pyramid is influenced by the application type.

Unit testing

Steve Sanderson wrote a great blog post about the cost and benefit of writing unit testing. I advise you to read it, as he makes great points in there. The summary of his post is that depending on what code you have, the costs and benefits are different.

The reasoning is as follows:

• Algorithmic code with little dependencies is easy to test because you have a fixed set of outcomes for a fixed set of inputs. It’s also very beneficial, because code tends to be non-obvious and mistakes are easily made.
• Trivial code is easy to test, but there’s little to no benefit in doing so since the chances of catching a mistake are slim
• Coordinators (code that “glues” together a bunch of dependencies) is difficult to test, because you’d need a lot of mock, stubs and fakes and when you change the implementation you usually have to change the test. There’s also very little value in testing it, because this code usually doesn’t really do something, it just delegates.
• Then there’s non-trivial code that has a lot of dependencies. This code is difficult to test, because there are a lot of dependencies, but it would be good to add some tests as the code can be non-obvious.

Taking this into account, when your application has a lot of algorithmic code, you should probably opt for a thicker layer of unit tests. In practice though, most of the applications I see have a majority of coordinators and trivial code (and sadly also overcomplicated code). So, unless you have a specific application that is very algorithmic in nature, I’d tend to write less unit tests. The objective should not be getting test coverage, the object is getting confidence when making changes to the software.

Integration testing

An integration test takes a few layers (or components or parts, whatever you want to call it) together and tests them as a whole. Depending on how your application is structured this can be easy or difficult. If you have an application that is very tightly coupled, it will probably be difficult to separate it from its UI. On the other hand, if you have an application that is easily configurable and flexible, it might be easier to isolate a part of it.
An example of this could be a Web API back-end with a Javascript front-end, hosted on OWIN with a database underneath. In order to just test the server part, you could self-host the Web API in your tests, use an in-memory SqLite database and run all your tests in-memory for a fast to write and run, deterministic suite of tests. In that case, put some more weight on the integration layer. If you have a legacy application that doesn’t lend itself to isolation, write less integration tests (or refactor it first).

UI testing

The UI of your application is what end users will see, so it’s very important to test. The three things holding us back from writing more UI tests are speed, brittleness and ease of writing. Let’s look at them one-by-one.

If you have a rather small application, even though your tests are relatively slow, the suite as a whole will not take very long and you can probably afford to cover most of it with UI tests. For a very large application, this will almost be impossible, unless you start parallelizing your tests (but that works against the third principle, ease of writing).

Brittleness of tests comes from an application not being deterministic under different circumstances. For example, a web app might behave differently when there’s a slow connection, a desktop app could behave differently when certain programs are installed or not.

Depending on how complex your UI is, writing UI tests can be more difficult or not. A UI where you have to go through ten different steps to execute a use case is harder to test then one where you have to do two steps. Other than that, the type of application often dictates which tools you can use. There are plenty of test tools for web applications, but very few can script interactions for a mobile application.

The amount of UI testing you want to do depends largely on the size of the application, how deterministic it is, how easy the use cases are and what tools are available.

The test-refactor cycle

The factors written above only work when you apply them in combination. You could easily find yourself in a situation where you have non-algorithmic code, that doesn’t lend itself to isolation, is big and has difficult use cases. You can’t just slim down all three layers of the pyramid. If you get into a situation like this, the application simply does not lend itself easily to automated testing. When that is the case, you could rely primarily on non-automated testing. This is a bad idea (why it is a bad idea deserves a whole different blog post) and should always be a temporary solution.

It’s better to refactor your code to make it more testable. That’s a catch 22, you can’t refactor because you need automated tests and you can’t write automated tests because you need to refactor first. The best approach here is to get into a test-refactor cycle. The idea is that you make the smallest possible refactor to allow you to write an automated test. When that test passes, you can refactor more (because now you have a test). If you keep doing this and focus on the attributes mentioned in the previous section, you’ll soon notice that you can get to a higher level of confidence.
Often it’s impossible to start these initial cycles on a unit test or integration level without doing a lot of refactoring upfront. The easiest way to do is, is through UI testing. When you have no tests, the first thing you need is confidence (the most important part of testing). UI testing will give you the most confidence and requires little to no refactoring upfront. Once you have that in place, you can start moving down the pyramid and isolate parts of the application for integration testing. When you built up some confidence through these tests, it’s time to refactor the lower level code and refactor the code into pure algorithms, coordinators and trivial code. The algorithms can then be unit tested.

Summary

The testing pyramid is a good starting point to structure your test efforts. We should skew the pyramid depending on the application though. Therefore, we should take into account various factors. To summarize, here are a few guidelines:

• Algorithmic code should be tested with unit tests
• Do not unit test code that is trivial or just delegates tasks
• Make your application configurable so you can isolate parts of it and test it with integration tests
• For legacy applications, start with UI tests, refactor and then add integration tests. Another round of refactor should expose the algorithms, which you can then unit test.

To illustrate my point, here are a few examples of applications and what their test distribution should be (the numbers are totally made up, it’s just to give an idea on where to put the emphasis):

1. A modern web app written in JavaScript, backed by a REST API, using Web API hosted on OWIN. The application uses Entity Framework as an ORM and serves to update an inventory of shopping items.
   1. Unit tests: 10%. There’s very little algorithmic code and you would be stubbing out the EF-layer which gives a false sense of security.
   2. Integration tests: 80%. You can self-host the API and attach an in-memory database. This will give you fast, reliable and fairly easy to write tests
   3. UI tests: 10%. Since our integration tests cover a fair part of our stack, the UI tests should just cover the main use cases.
2. A legacy web app, built with ASP.NET Web Forms that is used for listing and browsing properties (real estate).
   1. Unit tests: 5%. Web forms are notoriously hard to unit test because of their inherent dependencies. Extract the algorithms and put those under test. Leave the rest for higher layers.
   2. Integration testing: 5%. If there’s no possibility to bypass the UI, it will be very hard to write integration tests. Implement the part that can be bypassed
   3. UI testing: 90%. The application does not lend itself to other types of tests, because it wasn’t built with testability in mind. Your first job is to gain confidence in your refactors and changes to the code base. A UI test does not need refactoring and gives you a high level of confidence. Once you have that confidence, you can start refactoring the code and slowly add more weight to the Integration and Unit tests
3. A REST API for a bank, that allows you to send requests for calculating mortgages, loans and investments
   1. Unit tests: 80%. The code is highly algorithmic (and the algorithms are stable). You can write tests that don’t need a lot of maintenance, because an implementation detail won’t need a change to the test
   2. Integration tests: 20%. Since you have a REST API, you can easily write some tests that exercises the API and checks the result to see whether all units are composed correctly.
   3. UI tests: 0%. There is no UI and the integration tests are end-to-end tests.

These examples are quite arbitrary, but if you look at the application you’re working on, you’ll surely see some similarities and I hope by applying these techniques you can gain more confidence in your code.

The post The test pyramid appeared first on Kenneth Truyers.

If we want to test our website fully, we need to be able to test it on a variety of browsers, devices and screens. Building all this infrastructure is very costly and time consuming. But new companies such as BrowserStack and SauceLabs have a solution for that: they provide VM’s for all kinds of browsers, configurations and emulators. We can make use of their infrastructure by running our acceptance tests on their resources.

Another problem acceptance tests usually pose is that they are slow (in comparison to unit tests). By carefully maintaining your tests, you can get them as fast as they can possibly be, but they are slow by nature so there’s only so much you can do.

In this post I’ll work through an example on how to run a SpecFlow test remotely on multiple BrowserStack VM’s at the same time. These are the goals:

• No need to build/buy infrastructure
• Run tests simultaneously, so that adding configurations doesn’t slow down the process
• Make it easy to add configurations
• Make tests executable from the command-line, by anyone, at any time

1. The tests

As an example I’m going to use a simple test that executes a google search and then verifies whether results were returned:

Feature: Searching google

Given I am on the google page
When I search the web
Then I get search results

The step definitions are quite straightforward (intentionally so):

[Binding]
public class GoogleSteps
{
    readonly IWebDriver _driver;

    public GoogleSteps()
    {
        _driver = (IWebDriver)ScenarioContext.Current["driver"];
    }

    [Given(@"I am on the google page")]
    public void GivenIAmOnTheGooglePage()
    {
        _driver.Navigate().GoToUrl("http://www.google.com");
    }

    [When(@"I search the web")]
    public void WhenISearchTheWeb()
    {
        var q = _driver.FindElement(By.Name("q"));
        q.SendKeys("Kenneth Truyers");
        q.Submit();
    }

    [Then(@"I get search results")]
    public void ThenIGetSearchResults()
    {
        Assert.That(_driver.FindElement(By.Id("resultStats")).Text, Is.Not.Empty);
    }
}

Note that here I’m accessing the UI immediately in my step definitions. This is done for simplicity’s sake. In a real-word scenario, you probably want to use some indirection through the use of Page Objects. For more information on how to create maintainable specflow tests, you can refer to my article about Automated acceptance tests with Cucumber

2. The driver

To drive the browser I will be using Selenium (as seen in the example above). Selenium can drive the browser through an instance called a WebDriver. There are a few different WebDrivers available for IE, Chrome, FireFox, … but there’s also a driver called RemoteWebDriver. This is a driver that can drive a browser on a different machine. This is the one we will be using.

Nevertheless, we don’t want to tie any of our tests to a particular type of WebDriver. Luckily they all implement the IWebDriver interface.

2.1 Instantiating the driver

If we could use constructor injection, we could inject an instance of an IWebDriver into our step definitions. Unfortunately, SpecFlow doesn’t allow this. The next best thing we can do is use a Service Locator pattern. SpecFlow provides a Dictionary-like object called a ScenarioContext, which contains objects available to the current scenario. In the example above you can see that upon instantiation of the GoogleSteps-class, we get the driver from this dictionary. The next bit of code shows how we set up the driver at the beginning of a scenario:

[Binding]
public class Setup
{
    IWebDriver driver;

    [BeforeScenario]
    public void BeforeScenario()
    {
        driver = new FirefoxDriver();
        ScenarioContext.Current["driver"] = driver;
    }

    [AfterScenario]
    public void AfterScenario()
    {
        driver.Dispose();
    }
}

Through the use of some SpecFlow hooks, we create a driver before each scenario and tear it down after the scenario finishes. In this example we just created a local FireFox driver. At this point we can execute our test and it will run successfully in a local FireFox instance.

2.2 Configuring the driver to use browserstack

The next step is executing the same test, but on a remote machine managed by BrowserStack. To do this, we need to create a RemoteWebDriver, and configure it accordingly. When you create a RemoteWebDriver, you need to provide two arguments:

• The Url of where the RemoteDriver will accept commands (this is provided by BrowserStack)
• The capabilities: this is a loosely typed dictionary of parameters that will be sent to the RemoteDriver. In this case, BrowserStack dictates which parameters you need.

So first of all, let’s create an instance of the DesiredCapabilities-class and set the properties accordingly:

[BeforeScenario]
public void BeforeScenario()
{
    if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
        new Process
        {
            StartInfo = new ProcessStartInfo
            {
                FileName = "BrowserStackLocal.exe",
                Arguments = ConfigurationManager.AppSettings["browserstack.key"] + " -forcelocal"
            }
        }.Start();

    var capabilities = new DesiredCapabilities();

    capabilities.SetCapability(CapabilityType.Version, "33");
    capabilities.SetCapability("os", "windows");
    capabilities.SetCapability("os_version", "8");
    capabilities.SetCapability("browserName", "firefox");

    capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
    capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
    
    capabilities.SetCapability("project", "Google");
    capabilities.SetCapability("name", ScenarioContext.Current.ScenarioInfo.Title);

    capabilities.SetCapability("browserstack.local", true);

    driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
    driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
    ScenarioContext.Current["driver"] = driver;
}

I have separated the capabilities in to four different sections:

• The first four capabilities determine what browser and platform you want to run these tests on.
• The next two, indicate your BrowserStack username and key. These you can get from the BrowserStack interface after you create an “Automate”-account (or a trial)
• The last two items are optional and are merely so you can see what project and what test is running from the BrowserStack interface

With these capabilities set up, we can now create an instance of the RemoteWebDriver:

driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);

The driver is still of type IWebDriver, so we don’t need to change anything to our steps. Provided we have entered the correct username and key, we can now run our tests remotely on a VM managed by BrowserStack.

2.3 Allowing BrowserStack to access local resources

In the above example, I’m only accessing a public website. Chances are that in a real world scenario you will be testing a QA environment or even a dev environment on your PC. These are usually not publicly accessible. To allow BrowserStack access to these resources, they allow you to set up a tunnel through your PC and then access the resources from your PC. To set this up, you need to:

• Download the appropriate binary from their website (https://www.browserstack.com/local-testing#command-line)
• Run the binary providing your key as a command-line argument (ie: browserstacklocal.exe -<yourkeyhere>)
• Indicate that you need local access in the capabilities

So before we run our tests, we can run the executable from the scenario hook:

if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
    new Process 
    { 
        StartInfo = new ProcessStartInfo 
        { 
            FileName = "BrowserStackLocal.exe", 
            Arguments = ConfigurationManager.AppSettings["browserstack.key"] + " -forcelocal" 
        } 
    }.Start();

3. Running tests from the command-line

The tests can now be ran from Visual Studio. To run these tests from the command-line, we can use the unit test runner. In this case I have used NUnit as the underlying test framework, so I’ll be using the NUnit runners. To run the tests from the command line you run the following command:

nunit-console.exe /xml:nunit.xml /nologo /config:release <pathtodll>.dll

4. Running tests in parallel

Now that we have the tests running in BrowserStack from the command-line, it’s time to start running them simultaneously on various configurations. In order to run them on various configurations we need a few things:

• Parameterize the capabilities (namely os, os_version, browsername and version)
• Run several test at the same time with different parameters

There’s one problem with this approach: when we run NUnit through the console there’s no way to pass parameters to the tests. The general approach given on various forums is to set environment variables. Since we are running the tests simultaneously, this doesn’t help us. There’s one thing however that lets us parameterize how we run the tests: NUnit allows us to specify the configuration we want to run the tests with. So I have come up with the following approach:

• Use one solution configuration for every configuration we want to test
• Extract the os, os_version, browsername and version to the configuration
• Use config transforms to vary these parameters based on the solution configuration
• Use Powershell to run various instances of the NUnit-console process with different configurations

4.1 Creating different solution configurations

For each configuration we want to test, we will add a different solution configuration. Usually I start by deleting the standard release configuration and renaming the debug-configuration to something more sensible (eg: Win8Firefox33). You can add as many solution configurations as you want. It’s best to copy them from the original Debug-config as this will have all the settings necessary to be able to debug your code.

4.2 Extract the parameters to the configuration

We also need to change the capabilities to fetch these values from the config:

capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);
capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);

4.3 Add config transforms

When you have created the different solution configurations, you now need to add a config transform for each configuration. This is an example for my Win8Firefox33 configuration (note that the name is not important, it’s merely a convention)

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="firefox" xdt:Transform="Insert"/>
        <add key="os" value="windows" xdt:Transform="Insert"/>
        <add key="version" value="33" xdt:Transform="Insert"/>
        <add key="os_version" value="8" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

If your not sure how to add config transformations to a class-library, you can use a tool such as SlowCheetah to do it for you.

4.4 Run various instances of the NUnit-console process with Powershell

We can now already run a different configuration on BrowserStack by varying the config-parameter of the previous command. Here are two examples of how to run a different configuration:

nunit-console.exe /xml:nunit.xml /nologo /config:Win8Firefox33 <pathtodll>.dll 
nunit-console.exe /xml:nunit.xml /nologo /config:Win7Chrome38 <pathtodll>.dll 

The next step is executing all the configurations we have in parallel with powershell. To do this we’ll need to execute the following steps:

• Get all the configurations in the solution
• Compile the project in each configuration
• Run all configurations in parallel

To get all the configurations in the solution, we can use this small function:

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?<config>\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

This will open the .sln file, use a regex to find the configurations available, deduplicate them and return them as an array.

Next step is compiling the project against the different configurations. We will be using MsBuild for this:

@(Get-SolutionConfigurations "<path.to.sln>") | foreach {
    msbuild <path.to.csproj> /p:Configuration=$_ /nologo /verbosity:quiet
}

Now that the project is compiled in all configurations we can run the tests in parallel. For this, we will be using Powershell jobs:

@(Get-SolutionConfigurations "<path.to.sln>")| foreach {
    Start-Job -ScriptBlock {
        param($configuration)

        nunit-console.exe /xml:nunit_$configuration.xml /nologo /config:$configuration <path.to.dll.for.this.config>
    } -ArgumentList $_ 
}
Get-Job | Wait-Job
Get-Job | Receive-Job

This snippet retrieves the configurations and it start a job for each one. When all jobs are started it waits for all of them to complete and then receives and writes the output of all of them to the console one by one.

5. Reporting

Once we have run all our tests, we want to report the results. First of all, you can look at your tests performing live via the BrowserStack website. This is great and allows you visually verify any errors if the occur. It also shows you a complete log of everything that has happened in the scenario. The image below shows what you can see while the tests are running:

You can see in real-time which tests are running, how far they are and even follow on-screen what they are doing. That’s pretty awesome for live debugging and monitoring.

Apart from live debugging, we also need to create reports when tests have finished so we can check them afterwards or even archive them for later review. To do this, you can run the specflow tool:

specflow.exe nunitexecutionreport <path.to.csproj> /out:specresult.html /xmlTestResult:nunit.xml /testOutput:nunit.txt

When we add this to our parallel test runner, the script becomes as follows (we need to parameterize the text-files because we’ll have one for each configuration):

@(Get-SolutionConfigurations "<path.to.sln>")| foreach {
    Start-Job -ScriptBlock {
        param($configuration)

        try
        {
            nunit-console.exe /labels /out=nunit_$configuration.txt /xml:nunit_$configuration.xml /nologo /config:$configuration <path.to.dll.for.this.config>
        }
        finally
        {
            specflow.exe nunitexecutionreport <path.to.csproj> /out:specresult_$configuration.html /xmlTestResult:nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_
}
Get-Job | Wait-Job
Get-Job | Receive-Job

Now every NUnit test run will dump two files: an XML-file with the test results and a TXT-file with more information. The SpecFlow runner will then parse these files and generate an HTML-report.

6. Putting it all together

Now that we have gotten all steps finished, we can put everything together. To execute the tests we need the following:

• A parameterized setup of the driver before each scenario
• A config transformation for each configuration we want to run
• A PowerShell build script that can execute the configurations in parallel and report on them

For completeness, here is the full code for each of these requirements. Alternatively, you can clone the code from GitHub and play with it yourself.

6.1 Parameterized driver

[BeforeScenario]
public void BeforeScenario()
{
    if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
        new Process
        {
            StartInfo = new ProcessStartInfo
            {
                FileName = "BrowserStackLocal.exe",
                Arguments = ConfigurationManager.AppSettings["browserstack.key"] + " -forcelocal"
            }
        }.Start();

    var capabilities = new DesiredCapabilities();

    capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
    capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
    capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
    capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);
    
    capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
    capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
    capabilities.SetCapability("browserstack.local", true);
    
    capabilities.SetCapability("project", "Seatwave.Websites.Consumer");
    capabilities.SetCapability("name", ScenarioContext.Current.ScenarioInfo.Title);

    driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
    driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
    ScenarioContext.Current["driver"] = driver;
}

6.2 Config transformations

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="firefox" xdt:Transform="Insert"/>
        <add key="os" value="windows" xdt:Transform="Insert"/>
        <add key="version" value="33" xdt:Transform="Insert"/>
        <add key="os_version" value="8" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

6.3 Build script in Powershell

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?<config>\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

@(Get-SolutionConfigurations "<path.to.sln>") | foreach {
    iex { msbuild <path.to.csproj> /p:Configuration=$_ /nologo /verbosity:quiet }
}

@(Get-SolutionConfigurations "<path.to.sln>")| foreach {
    Start-Job -ScriptBlock {
        param($configuration)

        try
        {
            nunit-console.exe /labels /out=nunit_$configuration.txt /xml:nunit_$configuration.xml /nologo /config:$configuration <path.to.dll.for.this.config>
        }
        finally
        {
            specflow.exe nunitexecutionreport <path.to.csproj> /out:specresult_$configuration.html /xmlTestResult:nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_
}
Get-Job | Wait-Job
Get-Job | Receive-Job

7. Demo project

In the code above, I have omitted a few things for brevity’s sake. In order to be able to run msbuild, the nunit console runner and specflow, you need to ensure that all the paths are set up correctly (ie: added to your environment variables). Therefore, I have created a full working implementation, which you can find on GitHub.

To run it you need to do the following:

• Clone the repository to your PC
• Create a trial account on BrowserStack
• Go to Account => Automate and copy the UserName and Value into the app.config
• Open a command prompt and run “powershell –file build.ps1”

The post Running SpecFlow Acceptance Tests in parallel on BrowserStack appeared first on Kenneth Truyers.

• Aspect oriented programming versus composition in Simplify code by using composition
• Hand-written SQL or Micro-ORM’s versus ORM’s in How to ditch your ORM
• DI containers versus Pure DI in How to use Pure DI
• Partial application versus over-usage of interfaces in Using partial application for dependency injection

In this post I will reiterate these patterns by combining them all into a sample application. The application can be found on GitHub.

Disclaimer: the sample application serves to demonstrate patterns and practices. It is not intended to be fully tested nor complete. I do not pretend for this architecture to be the golden standard for development. I’m merely using it to show examples and alternative solutions to common and recurring problems I have faced. If you have any ideas or suggestions, feel free to comment, fork, throw a pull-request or engage with me on twitter (@kennethtruyers). The discovery and learning part is what interests me.

The sample app

The application is a rest API built with ASP.NET Web API (primarily because it saved me from writing a UI) on top of a SQL database. It allows you to create a user, update its profile and keep track of a list of friends. They key patterns that I want to illustrate are these:

• The separation of the read and write model.
• The use of an event-bus, query bus and command bus to interact between the endpoints, domain and the database.
• The use of partial application to do dependency injection.
• The use of composition to avoid attribute based programming.
• Testability

This is a short overview of the app structure:

• API: Contains the controllers. The controllers are very lightweight. They are just declarations of endpoints and dispatch either a query or a command
• App_start: Contains the regular start up code for a web api project and the the code that bootstraps the dependency resolution mechanism
• Application: The application is the part that coordinates all the input we receive to either the database or the domain.
• Domain: This contain all our custom logic. Since it’s a sample app, there’s not a lot there, but in a large application, this would be the beef of the app. It raises events about changes.
• Infrastructure: Contains PetaPoco as a Micro-ORM to talk to the database and a logging component to demonstrate composition
• ReadModel: Flat DTO’s that serve as output for query operations
• SimpleCode.Tests: Contains tests for the domain logic inside the User class

I’m not going to go over a the code, since most of the patterns are discussed in the previous posts and if you want you can have look at the sample application. I do want to highlight the main traits this app has.

Separation of read and write model

On the write-side, because our domain is free from read concerns, it allows our model to be very expressive. That means we don’t need public properties, but can expose only behavior. On the reading side, we can read directly from the database and project the data immediately into the read-model. This means that we don’t need any mapping between viewmodels and domain entities. A tool like AutoMapper is simply not necessary with this structure.

Event, Query and Command bus

The different buses are what composes the application while at the same decoupling components from each other. It allows for a clean separation between domain, database and the public endpoints. An extra benefit is that our application becomes scalable. The sample app consists of in-memory buses, but you could easily create a bus that communicates over a remote channel, thereby splitting up the read, write and endpoints into different physical tiers.

Partial Application and dependency injection

Because of the usage of partial application and pure DI as a mechanism for composing the application, we don’t need a DI-container. The entire application is configured inside the BootsTrap-method (WebApiConfig-class). A view on this method will tell you exactly how the application is composed. It frees us from learning the details about a specific container, increases readability and reduces complexity. The bootstrap-method can grow quickly once you start adding code to this project. But that’s OK, because that code has value, it tells you how the app works. It’s also simple code, because it’s plain object composition.

Another part of the equation is that we didn’t need to interfere with the way Controllers are constructed. Because the buses have public static methods, we can use them from anywhere. It allows our controllers to be mere declarations of endpoints (that are not at all served by unit tests) and place the dependency graphs one level lower. (see testability for more details about the ‘static’-ness of the buses).

Composition

Because we are using partial application, we can wrap different types of functions around other functions. This allows us to create cross-cutting concerns as separate elements and then apply them in the composition root (Bootstrap-method). An example of this can be seen where we wrap a logging-function around a handler function in the composition-root. This frees us of the burden of an AOP-framework and scattered attributes through our code base.

Testability

A concern that some people may have is that the Command, Query and Event bus are static classes with static methods and that they are not replaceable in tests. But that’s not an issue, we can actually just use them in our tests as well. If you look at the user test, you’ll see that we fire a few commands at the handler and then check whether we receive the correct events on the bus. Because commands and events are really domain concepts and should be tied closely to the problem domain, it makes for a nice way of testing. It let’s you write tests that say: If I create a user (command) then a user was created (event), which correlate closely with the domain language. We’re issuing commands and then testing whether that command had the correct reaction of the system (events).

Conclusion

The combination of these patterns allows us to create an architecture without depending on external tools. Because external tools introduce complexity, such an architecture yields simpler code. I want to stress again that when I say simple, I do not mean easy. The code this app showcases is not really easy (easy as in, a junior could write this first day out of school). This code is not easy, but it is simple in the fact that there is no magic involved by frameworks (generating code at runtime, dynamically resolving factories, reflecting over properties to map from one type to another, …). If a bug would show up in this code, it will definitely be a visible one and troubleshooting will be rather simple.

This code is also not an endpoint, there are still things that I don’t like about it as well:

• The bootstrap method is not testable, that means that you could encounter an error at runtime where a handler is not resolved. For commands and events, this could be done through a unit test, by reflecting over all classes that implement ICommand or IEvent and trying to resolve a handler (although it’s rather ugly). For the query handlers it’s more difficult as you could theoretically dispatch any combination of a requested result by any type of query. I’m not sure how this could be made testable without an insane amount of reflection code (any ideas here are more than welcome). 
• There’s still quite a bit of ugly syntax overhead. If you look at the command handlers, they’re signature is public static readonly Action<T> NameOfHandler = command => . That’s a lot of words to say you’re declaring an Action<T>, but I guess that is just how C# syntax works. F# would probably offer a huge improvement in this area.

As I said, I look at this code more as an exploration of interesting patterns in order to reduce complexity. If you have any feedback, suggestions or improvements, feel free to contact me. Happy to discuss, defend my opinions and/or admit I’m wrong.

Browse or clone the code on the GitHub

The post Simple code: a sample app appeared first on Kenneth Truyers.

public class TaskCommandHandlers : ICommandHandler<CreateTask>,
                                   ICommandHandler<MarkTaskDone>
{
    ITaskRepository _repo;
    INotificationService _notificationService
    public TaskService(ITaskRepository taskRepository, INotificationService notificationService)
    {
        _repo = taskRepository;
        _notificationService = notificationService;
    }
 
    public void Handle(CreateTask command)
    {
        repo.Insert(new Task(command.Name));
    }
 
    public void Handle(MarkTaskDone command)
    {
        var task = _repo.Get(command.TaskId);
        task.Status = TaskStatus.Done;
        task.ModifiedBy = command.UserName;
        _repo.Update(task);
        _notificationService.NotifyUser(command.UserName);
    }
}

What we did was combining the different parameters so that we have a common interface. This allows us to compose our code in different ways using the decorator pattern. This has certain advantages as discussed in the post, but there are still a few things that I don’t like about this code:

• There’s a lot of code that doesn’t really add value. In fact only 8 lines (the method bodies) out of a total of 25 are relevant. On top of that we have an interface (ICommandHandler).
• The dependencies are scoped incorrectly. They are not really dependencies for the entire class but localized to the separate methods. (Although the repo is used in both methods, in any instance of this class, conceptually it belongs to each of the methods separately.)
• A class has state and behavior, but this class does not really have any state apart from the dependencies, which are wrongly scoped.

To solve this, we could pass the dependencies into the methods instead of in the constructor:

public class TaskCommandHandlers : ICommandHandler<CreateTask>,
                                   ICommandHandler<MarkTaskDone>
{ 
    public void Handle(CreateTask command, ITaskRepository repo)
    {
        repo.Insert(new Task(command.Name));
    }
 
    public void Handle(MarkTaskDone command, ITaskRepository repo, INotificationService notificationService)
    {
        var task = repo.Get(command.TaskId);
        task.Status = TaskStatus.Done;
        task.ModifiedBy = command.UserName;
        repo.Update(task);
        notificationService.NotifyUser(command.UserName);
    }
}

This already looks a bit better, but there’s a problem: now we don’t have a common interface anymore. To solve this, we could apply the same technique as we did in getting to the common interface ie:  combining the parameters into a new type. That wouldn’t be a good idea though. Instead, let’s first move away from the class-behavior and declare these code block as static Action‘s:

public class TaskCommandHandlers
{
    public static readonly Action<CreateTask, ITaskRepository> CreateNewTask = 
        (command, repo) => repo.Insert(new Task(command.Name));

    public static readonly Action<MarkTaskDone, ITaskRepository, INotificationService> MarkTaskAsDone = 
        (command, repo, notificationService) =>
        {
            var task = repo.Get(command.TaskId);
            task.Status = TaskStatus.Done;
            task.ModifiedBy = command.UserName;
            repo.Update(task);
            notificationService.NotifyUser(command.UserName);
        };
}

In this code:

• There’s no constructor
• There are no interfaces
• Dependencies are correctly scoped
• We’ve reduced the ratio of important lines/overhead to 8/14 instead of 8/25.

There’s still the issue of a non-conforming interface though. At some point we do need to call these methods and we’d like to pass in just the command and have the other dependencies resolved automatically. Our client needs either an Action<CreateTask> or an Action<MarkTaskDone>, but what we have is an Action<CreateTask, ITaskRepository> and an Action<MarkTaskDone, ITaskRepository, INotificationService>. For this we are going to use partial application.

Partial application explained

Before I show how to do this, I want to give a short explanation of what partial application is. Partial application is a concept that stems from Functional Programming (FP), but since C# has some elements of FP, we can use it here as well.

partial application (or partial function application) refers to the process of fixing a number of arguments to a function, producing another function of smaller arity (source: wikipedia)

Here we can see a sample of partial application in action:

Func<int, int, int> sum = (a, b) => a + b;

Func<int, int> add10To = a => sum(a, 10);

sum(5, 10);

add10To(5);

In the above example you can see how we create a Func (sum) which accepts two ints and returns an int. On the second line, we create a Func called add10To, which accepts a single int and returns an int. Inside that Func, we call the original function, with the single parameter and the value 10. What we have done is fixed one of the arguments of the original function to a fixed value. This is exactly what we need to solve our problem above.

Partial application as a DI mechanism

Knowing what partial application is, we can now create an Action<CreateTask> as follows:

var repository = new TaskRepository();
var notificationService = new NotificationService();

Action<CreateTask> create = command => 
    TaskCommandHandlers.CreateNewTask(command, repository);

Action<MarkTaskDone markDone = command =>
    TaskCommandHandlers.MarkTaskAsDone(command, repository, notificationService);

Partial application in the composition root

To be able to resolve these actions we need to declare them in our composition root. Let’s look at how we would do that in an ASP.NET web API project. Because we don’t want to inject multiple Action<T>‘s in our controllers, we will be creating a dispatcher with static methods. A controller will look like this:

public class TaskController : ApiController
{
    [HttpPost]
    public void CreateTask(CreateTask command)
    {
        Commands.Dispatch(command);
    }

    [HttpPost]
    public void MarkTaskAsDone(MarkTaskDone command)
    {
        Commands.Dispatch(command);
    }
}

The Commands-class is a static class with public static methods (see below). The TaskController is in fact just a regular controller with an empty default constructor. We won’t ever need to inject anything in here, because the only thing a controller does is declare endpoints and from these endpoints we dispatch commands. For query operations, we can follow the same pattern as with the commands, but I’ve omitted that part for the sake of brevity.

Note about testing: Some will argue that you can’t substitute the dispatcher in unit tests. Personally, I wouldn’t test the controller. All you would be testing is whether it dispatches the command that you pass in, which is not a very valuable test.

The dispatcher needs to be able to register handlers and dispatch commands to the correct handlers:

public class Commands
{
    static readonly Dictionary<Type, Action<ICommand>> handlers = 
	new Dictionary<Type, Action<ICommand>>();
    public static void Register<T>(Action<T> handler) where T : ICommand
    {
        handlers.Add(typeof(T), x => handler((T) x));
    }

    public static void Dispatch(ICommand command)
    {
        handlers[command.GetType()](command);
    }
}

In the register-method we save all handlers in a dictionary, with the key based on their type. When we’re dispatching a command, we look it up in the dictionary and call the action. Our composition root can then be as follows:

public class CompositionRoot
{ 
    public static void Bootstrap()
    {      
        // Create the services we need         
        var repository = new Lazy<TaskRepository>(new TaskRepository());   // use lazy here, because it’s expensive to construct (suggestion by FizixMan on Reddit)
        var notificationService = new NotificationService();
          
        // Use partial application to fix the extra parameters the handlers need
        // and create and register an Action<CreateTask> and an Action<MarkTaskDone>
        Commands.Register<CreateTask>(createTask => 
            TaskCommandHandlers.CreateNewTask(createTask, repository.Value);

        Commands.Register<MarkTaskDone>(markAsDone => 
            TaskCommandHandlers.MarkTaskAsDone(markAsDone, repository.Value, notificationService));
    }
 }

In the composition root we create and register the handlers we’re going to need. This is where we close the extra parameters that the handlers need and turn them into plain Action<T>’s. We only need to call the Bootstrap method once when the application starts. This frees us from having to deal with Web API-specific code and this composition root is in fact portable to any other type of project.

Managing lifetime with Partial Application

A common treat of most containers is the ability to manage the lifecycle of a dependency. With partial application, this is built-in and it allows you to create custom lifecycles as well. There’s no need to learn special syntax, it’s just regular C#. To explain lifetime management, I’m going to use a slightly different graph:

Handler
    - CreateTask
    - Repository
    - NotificationService
        - Repository
    - SomeService
        - Repository

In the following examples I will focus on the lifetime management of the repository.

Singleton

For a singleton we need to instantiate the repository inside the Bootstrap method, but outside of the actual handlers, so it will only get instantiated once (when the method is called, on start up). Here the repository is created only once and is then reused every time we invoke the action.

public static void BootsTrap()
{                
    var repository = new TaskRepository();
    var notificationService = new NotificationService(repository);
    var someService = new someService(repository);
          
    Commands.Register<CreateTask>(createTask => 
        TaskCommandHandlers.CreateNewTask(createTask, repository, notificationService, someService);
}

Transient

Here every time the action is invoked, all services are instantiated.

public static void BootsTrap()
{               
    dispatcher.Register<CreateTask>(createTask => 
        TaskCommandHandlers.CreateNewTask(createTask, 
                                        new TaskRepository(), 
                                        new NotificationService(new TaskRepository()), 
                                        new SomeService(new TaskRepository());
}

Per request

Notice the subtle difference with the singleton: the services are created inside the Action<T> instead of in the method itself. So, for every time we call the Action, we’ll create a new the repository, but that instance will be shared by all services. (NOTE: Technically this is not per request, if not per graph, but since we will model our controller to send a single command per request, in practice it will be the same. You could create a pure per request lifetime by using the HttpContext.Current.Items bag, but I’ll leave that as an exercise )

public static void BootsTrap()
{               
    Commands.Register<CreateTask>(createTask => 
    {
        var repository = new TaskRepository();
        var notificationService = new NotificationService(repository);
        var someService = new SomeService(repository);
        TaskCommandHandlers.CreateNewTask(createTask, 
                                        repository, 
                                        notificationService, 
                                        someService);

    };
}

Custom lifetime

Since we’re using pure C#, we can mix and match any instantiation policy we want. In the following (contrived) example:

• notificationService and someService have a per request lifetime.
• notificationService and someService share the same singleton instance of the repository.
• The handler gets a new repository every time (transient).

public static void BootsTrap()
{               
    var repository = new TaskRepository();
    Commands.Register<CreateTask>(createTask => 
    {
        var notificationService = new NotificationService(repository);
        var someService = new SomeService(repository);
        TaskCommandHandlers.CreateNewTask(createTask, 
                                        new TaskRepository(), 
                                        notificationService, 
                                        someService);

    };
}

Composition

In my post simplify code by using composition I showed how to move from attribute based programming to a more flexible composition based programming. This composability is not lost when we use partial application instead of constructor injection. To add logging to the CreateTask handler, we create a new handler:

class LoggingHandlers
{
    public static Action<ICommand, Action<ICommand>> Log = 
        (command, next) => 
        {
            Log(command); // do custom logging here
            next(command);
        } 
}

Now we use partial application again to plug this in from within the composition root:

Commands.Register<CreateTask>(createTask => 
    LoggingHandlers.Log(createTask, createTask => 
        TaskCommandHandlers.CreateNewTask(createTask, repository));

Conclusion

Partial application is a very flexible and powerful pattern. When applied to DI, it focuses the attention on code that matters, correctly scopes dependencies and removes the cruft that constructor injection sometimes brings with it. It allows the same usage patterns of a DI-container with regards to lifetime management and it does not inhibit composability.

Update after comments on Reddit by FizixMan:

• The static Actions on the handler should be readonly, you don’t want other code to overwrite those
• In case construction of a dependency is costly, you can consider using a Lazy<T> to speed up application start up.

The post Using partial application for dependency injection appeared first on Kenneth Truyers.

DI-containers are beneficial if you have a complex application where you can rely on convention over configuration. If your application is not complex (and you should strive for that) or does not rely on conventions, a simpler approach can be followed by using pure DI. Before I dive in on how we can do this, let’s first iterate over a few of the disadvantages a container brings with it:

• Complexity: When you configure a container in your composition root, you will usually be relying on conventions to make sure that the right types are resolved at runtime. How simple or complex this is, is directly proportional to how simple or complex your conventions are. If conventions are very clear, it will be relatively easy to spot how a dependency will be resolved at runtime. If not, it will be difficult to see how the application is composed.
• Compiler assistance: Since you’re resolving your dependencies at runtime, the compiler can’t assist you. If you forget to declare a service it will just fail at runtime. This does not only go for registration but also for lifetimes. You can introduce lifetime mistakes and the compiler won’t be able to help you. An example of this is captive dependency (more information see: Captive dependency by Mark Seemann)
• Learning curve: Every container has a different API. In order to use it effectively, you need to have quite a good knowledge of that API. With so many containers available, it’s possible you’ll encounter different containers in different projects, which means you need to learn a new API once in a while.

The situation where you might want to choose a pure DI approach over a container approach is when you do explicit registration instead of convention over configuration or when your conventions are really complex.

Pure DI in ASP.NET Web API

In order to use pure DI, you need to build your dependency graph whenever you code is being called. In an ASP.NET Web API application, this is when your controller is constructed. To intercept at this point, you need to implement the IHttpControllerActivator interface:

public class CompositionRoot : IHttpControllerActivator
{
    public IHttpController Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
    {
        // Resolve controller of Type here
    }
}

Inside this method, you will be passed the requested controller-type and you need to return an instance of that controller. A few things you need to know about this class:

• You need to register it on application start up
      ( GlobalConfiguration.Configuration.Services.Replace(typeof(IHttpControllerActivator), new CompositionRoot() );
• This class will only be instantiated once and will be reused for the entire lifetime of the application
• The create-method will be called once per request

With this information, we can now infer how should configure lifetime management:

• Singleton instances need to be instantiated in the constructor
• Per request instances need to be instantiated inside the method
• Transient instances need to be instantiated whenever necessary
• Any custom lifetime is easily configurable by instantiating it when necessary

Let’s see how this looks:

public class SomeCompositionRoot : IHttpControllerActivator
{
    readonly SomeService singleton;
 
    public SomeCompositionRoot()
    {
        singleton = new SomeService(new SomeOtherSingletonService());
    }
 
    public IHttpController Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
    {
        var customLifeTime = new SomeCustomLifeTimeService();
        var perRequestService = new SomePerRequestService(customLifeTime);
 
        if(controllerType == typeof(MyController))
        {
            return new MyController(singleton,
                                    new TransientService(singleton, new SomeCustomLifeTimeService()),
                                    perRequestService,
                                    new SomeOtherTransientService(singleton, perRequestService);
        }
  
        throw new ArgumentException("Unexpected type!", "controllerType");
    }
}

Pure DI in a console application

In a console application, we don’t have the per-request lifetime, but the same principles apply. We need to construct the graph on application start, right inside void main (or extract it to a separate class):

public class Program
{   
    public static void main()
    {
        var customLifeTime = new SomeCustomLifeTimeService();
        var singleton = new SomeService(customLifeTime);
 
        var entryPoint = new EntrypointClass(new TransientDependency(singleton),
                                             singleton,
                                             new OtherTransientDependency(customLifeTime),
                                             new SomeCustomLifeTimeService());
        entryPoint.Run(); // or something similar
    }
}

This is obviously a bit more typing work than registering services with a container, but it has certain advantages:

• Compile-time safety: You cannot forget to register a service, because the compiler will tell you.
• Lifetime configuration: You cannot make a lifetime mistake such as captive dependency. Because you are declaring the singletons in the constructor you can see that the services that get passed in their constructors will also be singletons. Again this makes it easier to read your configuration.
• Explicit: Since you have very clearly marked places to declare singletons, per request and transient services, it makes the configuration very explicit. All types are also explicitly constructed so you can easily spot how the application is composed.
• Learning curve: This is just object composition, so you don’t need to learn the specific API of a container.

Conclusion

Pure DI leads to more explicit code. You’ll write more code, but again I apply the same mantra: it’s more code, but it’s simpler code, so I’m happy to type a bit more for the sake of simplicity. One part that this approach does not solve is another pet peeve of mine: the over usage of constructor injection. In a next post I’ll show an example of how to use partial application to tackle this problem.

The post How to use Pure DI appeared first on Kenneth Truyers.

ORM’s are not evil, they have certain advantages and disadvantages. These are some of their characteristics and how they may influence a project.

• Simplicity: At the start of a project, an ORM is a real productivity booster, because you can load and save objects by writing very little code. There’s probably very little complexity in your domain model, so your model is very similar to your database structure and so mapping is very easy. When your model becomes more complex, mapping will get more complex. When this happens, you have a problem: either your complex domain model will be restricted by how your database is designed (in order to have simpler mapping) or your mapping will become very complex. (Here’s one example of how an ORM can limit your ability to model your application: http://stackoverflow.com/questions/17275030/how-to-map-a-value-type-which-has-a-reference-to-an-entity )
• Abstraction: An ORM provides an abstraction. This abstraction is leaky. If you look at documentation of any ORM, you will find a lot of references to SQL concepts. I have never been able to treat the ORM as just an object store, every time I needed to know how the ORM does things in order to get the correct data. Think about it like this: if you wouldn’t know anything about SQL, would you be able to use an ORM?
• Learning curve: Every ORM has a different API. That means that with every ORM, you’ll have a new learning curve. Since it’s a leaky abstraction, it doesn’t free you from learning SQL either so now you not only need to learn SQL, but also (N)Hibernate and later Entity Framework and later …
• Efficiency: All ORM’s do admit that you will be giving up a bit of efficiency. For small projects, that’s not an issue. When it becomes an issue, you’ll need to bypass the ORM and access the database with plain SQL, again asserting the point that an ORM is a leaky abstraction. (See also the part on ORM’s in my post about simplicity in software)

So, ORM’s have certain disadvantages and in my opinion they are not a good fit for complex applications, because they tend to increase complexity. But they’re not useless either: they can increase productivity at the start of a project (and can be removed/replaced when necessary). If your application is small and is very CRUD oriented, they provide great value as well.

The reason that ORM’s can only provide a leaky abstraction is that object relational mapping is in fact very hard (also known as the Vietnam of Computer Science).

The problem

Object hierarchies are inherently very different from relational hierarchies. Relational hierarchies center around data, whereas objects gravitate towards behavior (at least it should). OO modeling is a lot more powerful than relational modeling and because software development is in fact very difficult, we want the most powerful tool at our disposal. The issue is that we try to create a mapping between a database and an object model. This has some consequences: we will either have a limited object model that is just a representation of our relational model (lowest common denominator), or we will have very complex mappings (which can break down as we continue to model).

Changing the problem

I’m not pretending that you should write yet another ORM, or that I’m creating a new revolutionary ORM. As I said, object relational mapping is hard, so instead of trying to solve this problem, we want to change the problem so we don’t have to deal with it.

A first step towards changing the problem is realizing that reading and writing are two very different operations. Typically when you write, you want to ensure consistency. To ensure consistency, you need a strong model (DDD is one approach towards a stronger model). When you read, you’re trying to display the saved data in a certain form. Two read operations on the same data, may want a different representation. Ideally, when you’re reading you want a simple, flat model. Thus, the requirements imposed on the model are different for reading and writing. If we create a model that caters to reading and writing it will be more complex.

The first step towards easier mapping is to split out our read and write model. This means we can have our simple models on the read side, but still have a strong model that ensures consistency on the write side.

Tackling the read side

On the read side models are relatively simple, so we don’t need any complex mapping. In fact, reading should just be about projecting data into our models. Because it’s just a projection, no ORM is needed and you can just write plain SQL queries. You could use one of the micro-ORM’s available (PetaPoco is my personal favorite on the .NET platform). Although technically these are also ORM’s, they don’t carry the same weight as their full-blown counterparts. The biggest difference is that they don’t try to abstract the database away. I prefer to think of them as SQL-libraries rather than ORM’s.

Tackling the write side

On the write side, we’ll usually have a complex model that enforces constraints. If we want to persist our entire entity (or aggregate root) at once, that means that we need to do some complex mapping or write complex queries. If you were to use a repository pattern, when you call the save method, that repository will somehow have to find out what has changed and how that relates to what is in the database. This is hard and is the root of most complex mapping. If these notifications are fine grained, the listeners can be very simple. Let’s see an example of how we would persist a user;

Example 1: Persisting a user using a repository

public class User
{
    public int Id {get; set;}
    public string Name {get; set;}
    public List<User> Friends {get; set;}
}

public class UserRepository
{
    public void Save(User user)
    {
         // what has changed?
         // added, removed friends?
         // updated name of a friend
         // changed name?
    }
}

In the example above, when we save a user, so many things could have changed that the save-method has a hard time figuring out what to persist. An ORM takes this work out of your hands but then your mapping can become complex (is it a many-to-many? what happens if I change a friends name? if a friend does not have an ID, does it do an insert?, …). This is a trivial example and most ORM’s can handle this fairly easy, but in more complex scenarios, mapping can become really difficult and obscure. Handling this manually is very difficult as well, since you need to manage all these changes yourself (and essentially you’d be writing your own ORM).

In order to circumvent this complexity, we need a different strategy. We could let the model notify what has happened and then have a dedicated listener listen to those changes:

public class User
{   
    int id;
    public void AddFriend(User friend)   
    {
        EventBus.Raise(new FriendAddedToUser(id, friend));
    }

    public void RemoveFriend(User friend)   
    {
        EventBus.Raise(new FriendRemovedFromUser(id, friend));
    }

    public void ChangeName(string name)   
    {
        EventBus.Raise(new UserNameChanged(id, name));
    }
}

public class UserEventHandlers
{
    public void Handle(FriendAddedToUser @event)
    {
         // pseudo code
         // insert into user_friends (userid, friendid) values (@event.Id, @event.Friend.Id);
    }
    public void Handle(FriendRemovedFromUser @event)
    {
         // pseudo code
         // delete from user_friends where userid = @event.Id and @event.Friend.Id);

    }
    public void Handle(UserNameChanged @event)
    {
         // pseudo code
         // Update users set name = @event.Name where id = @event.Id;
    }
}

The User-class does not have any public properties, only method calls. We don’t need properties, since we’re not using this class to read and we’re not using an ORM. Whenever a method is called, the User class will push some event onto a bus. The bus will then look up one or more handlers to handle those events. In this case there’s just one for each event. Because these events are very fine grained, the resulting SQL is very easy to write (again you could use micro ORM to make life simpler, PetaPoco has an excellent SQL-builder that makes this trivial. Did I say I like PetaPoco yet?).
An added advantage is that it makes it easier to enforce constraints on your model. Here the methods just raised an event, but they could do anything to enforce invariants. In the first sample, public properties are exposed so there’s no way to control what happens. Of course you could add some checks, but if you’re going to a use an ORM, you will need public properties. You do need a bit of infrastructure code to set up the event-bus, but it’s fairly trivial and it’s a one-off investment.

We’re replacing a repository pattern with a publish-subscribe pattern in order to decouple domain logic from data access logic.

One part that I didn’t include here, is loading a user. This is a different use case than reading for displaying a user. In order to be able to enforce invariants, you need the whole aggregate to be loaded. Let’s say that a user can have a maximum of 10 friends. In order to enforce this, the method AddFriend needs to know how many friends there currently are. For that you can use the memento pattern:

Memento pattern: Without violating encapsulation, capture and externalize an object’s internal state so that the object can be restored to this state later. (Design patterns, elements of reusable Object-Oriented software)

public class User
{   
    int id;
    List<User> friends;
    public void AddFriend(User friend)   
    {
        if(friends.Count() < 10)
        {
            friends.Add(friend);
            EventBus.Raise(new FriendAddedToUser(id, friend));
        }
        else
        {
             throw new Exception("nope");
        }
    }

    public void RemoveFriend(User friend)   
    {
        EventBus.Raise(new FriendRemovedFromUser(id, friend));
    }

    public void ChangeName(string name)   
    {
        EventBus.Raise(new UserNameChanged(id, name));
    }
    public static User FromMemento(UserMemento memento)
    {
        var user = new User();
        user.id = memento.id;
        user.friends = memento.friends;
        return user;
    }
}

In this case, we’re using only half of the memento pattern (the restoring part) since the data is already captured in the database. Since the memento object is a simple bag of properties, it can easily be read from the database in the same way as we are projecting data into our read-model.

Conclusion

Using these patterns comes with the cost of more infrastructure code, although this code is relatively easy to write. In my experience, on complex domains it certainly pays off. In simpler domains, it’s often not worth the overhead. Object relation mapping is hard. ORM’s help solving this, but they come with the cost of complexity. In order to reduce complexity we can eliminate the ORM, but this is only possible if we can eliminate the object relational mapping. By leveraging some well-known patterns, we can simplify our database access. This allows for a rich write model and a light read model, while still allowing us to free domain logic from data persistence concerns.

The post How to ditch your ORM appeared first on Kenneth Truyers.