NUnit vs. XUnit vs. MSTest: Unit Testing Frameworks
Himanshu Sheth
Posted On: March 22, 2021
427059 Views
24 Min Read
One of the most challenging things to do is ‘making the right choice.’ Arriving at a decision becomes even more complicated when there are multiple options in front of you☺. The same is the case with choosing a testing framework for .NET Core. The three major C# Unit testing frameworks are MSTest, NUnit, and xUnit.Net. You should select the most appropriate test framework that suits your project requirements. In this blog, we will see a detailed comparison between NUnit vs. XUnit vs. MSTest.
We have earlier covered all the three frameworks independently and recommend you to refer to those articles for detailed information on these test frameworks. In this article, we will briefly go through the features and compare the frameworks (NUnit vs. XUnit vs. MSTest) so that you can arrive at a decision to select the best-suited framework.
These frameworks employ annotations or attributes to guide how the source code is interpreted. Attributes serve as meta-tags, providing additional information about the test methods and classes within the source code.
We would be demonstrating NUnit vs. XUnit vs. MSTest comparison using cross browser testing using the Selenium uses. When choosing a test framework, you should check the in-house expertise and check whether the framework is well-suited for the ‘type’ of project your team is working on!
TABLE OF CONTENT
NUnit
NUnit is an open-source testing framework ported from JUnit. The latest version of NUnit is NUnit3 that is rewritten with many new features and has support for a wide range of .NET platforms.
NUnit has been downloaded more than 126 million times from NuGet.org. This shows the popularity of NUnit within the .NET user community. As of writing this article, there were close to 24,000 questions tagged as NUnit on Stackoverflow. It is a popular test framework used for test automation. If you are planning to perform Test-Driven Development (TDD) with C#, you should have a close look at the NUnit framework. If you’re looking to improve your NUnit interview skills, check out our curated list of NUnit interview questions and answers.
Automation testing in NUnit can be performed using the Visual Studio GUI and the NUnit console (NUnit.ConsoleRunner). NUnit framework uses an attribute/annotation style system like other test frameworks; shown below are the most popular NUnit attributes used for Selenium automation testing.
| [SetUp] | There should be at least one method per test class. Marks a method that should be called before each test method. |
| [TearDown] | There should be at least one method per test class. Marks a method that should be called after each test method. |
| [TestFixture] | Marks a class that contains tests. |
| [Test] | Marks a method, i.e., an actual test case in the test class. |
| [TestCase] | Marks a method with parameters and provides the inline arguments. |
| [TestFixtureSetUp] | Marks a method that is executed once before the execution of any test method in that fixture. |
| [TestFixtureTearDown] | Marks a method that is executed after the last test method in that fixture has finished execution. |
| [Ignore] | Marks a test method or test class that should not be considered for execution, i.e., it is ignored. |
| [Category] | Specify the category for the test |
| [Indicates] | Marks that the test should be skipped unless it is explicitly run. |
| [OneTimeSetUp] | Methods that should be called before any of the child tests are executed. |
| [OneTimeTearDown] | Methods that should be called after the child tests are executed. |
| [Culture] | Indicates cultures for which a test or test fixture should be executed. |
| [MaxTime] | Maximum time frame in milliseconds for the successful execution of a method. A method is marked as a fail if the execution does not complete within MaxTime. |
| [Platform] | Platforms on which the test execution should be performed. |
| [TimeOut] | Timeout value in milliseconds for the test cases under execution. |
| [Retry] | Test will re-run if it fails till the time it is either successful or the maximum number of times have reached. |
| [Repeat] | Specifies the method that needs to be executed/repeated number of times. |
| [ExpectedException(ExceptionType)] | Marks a test that is expected to raise a particular exception, i.e., ExceptionType. |
| [Random] | Specifies generation of random values that will be passed as arguments to the parameterized tests. |
The attributes [Test] and [TestCase] are not extensible as those attributes are sealed.
Using NUnit framework, tests can be executed serially as well as parallel. Parallel test execution is possible at assembly, class, or method level. When parallelism is enabled, by default four tests can be executed in parallel. As Parallel test execution is supported in NUnit, it becomes a compelling option for Selenium automation testing and cross browser testing.
This NUnit Tutorial for beginners and professionals will help you learn how to use NUnit framework with Selenium C# for performing Selenium automation testing.
XUnit
xUnit.Net is an open-source testing framework based on the .NET framework. ‘x’ stands for the programming language, e.g., JUnit, NUnit, etc. The creators of NUnit created xUnit as they wanted to build a better framework rather than adding incremental features to the NUnit framework.
xUnit is created with more focus on the community; hence it is easy to expand upon. You can download xUnit from NuGet gallery. So far, there are close to 7,500 questions tagged as xUnit on Stackoverflow.
Below are some of the core reasons why the creators of NUnit decided to reconsider the framework and built xUnit.Net.
As of writing this article, the latest version of xUnit was 2.4.1. The framework follows a unique style of testing, and tags like [Test] & [TestFixture], a part of the NUnit framework, are no longer used in the xUnit framework.
The popular attributes [SetUp] and [TearDown] are also not a part of the xUnit framework. As per the creators of NUnit (and xUnit), usage of [SetUp] and [TearDown] led to code duplication, and they wanted to implement the same features in a much more optimized manner in xUnit. For initialization, constructor of the test class is used, whereas, for de-initialization, IDisposable interface is used. This also encourages writing much cleaner tests. This makes this C# unit testing framework a much better option when it comes to Selenium automation testing as it is more robust and extensible.
As far as NUnit vs. XUnit vs. MSTest is concerned, the biggest difference between xUnit and the other two test frameworks (NUnit and MSTest) is that xUnit is much more extensible when compared to NUnit and MSTest. The [Fact] attribute is used instead of the [Test] attribute. Non-parameterized tests are implemented under the [Fact] attribute, whereas the [Theory] attribute is used if you plan to use parameterized tests.
In NUnit and MSTest, the class that contains the tests is under the [TestClass] attribute. This was not a very robust approach hence [TestClass] attribute was also removed in xUnit. Instead, intelligence is built in the xUnit framework so that it can locate the test methods, irrespective of the location of the tests.
Shown below are the most popular attributes/annotations used in the xUnit framework:
| [Fact] | Marks a test method, i.e., actual test in a class |
| Assert.Throws Record Exception | Verify the raise and raise assert, irrespective of the place in the code where the problem occurs. |
| Constructor | This is not an attribute but is an ideal replacement for the [SetUp] attribute. The constructor should be parameter-less. |
| IDisposable.Dispose | This is not an attribute but is an ideal replacement for the [TearDown] attribute. This is where the code for performing necessary cleanup and de-initialization is included.
The decision to do away with [TearDown] was made as the investors of xUnit felt that a lot of unnecessary code was run before every single test execution. |
| [Trait] | Used to set arbitrary meta-data on a test |
| [Theory] | This attribute is used when data-driven tests have to be executed. In such cases, [Theory] has to be used instead of [Fact] attribute |
| [InlineData] | This attribute is used along with the [Theory] attribute to supply a subset of data against which parameterized tests will be executed. |
| [ClassData] | This attribute is used when the parameters being passed to the [Theory] tests are not constants. [Theory] [ClassData(typeof(some-data))] |
| [MemberData] | This attribute can be used to fetch data for [Theory] from a static method. The most common approach is to load the data from the property of a test class, i.e., using IEnumerable<: object[] > |
This xUnit Tutorial for beginners and professionals will help you learn how to use xUnit framework with Selenium C# for performing Selenium automation testing.
MSTest
MSTest is the default test framework that is shipped along with Visual Studio. The initial version of MSTest (V1) was not open-source; however, MSTest V2 is open-source. The project is hosted on GitHub. Like other test frameworks, it can also be used for data driven testing. You can download MSTest V2 from Nuget.org.
As of writing this article, there are close to 10,000 questions with the tag MSTest on Stackoverflow. Below are the most commonly used MSTest attributes.
| [TestInitialize] | Marks a method that should be called before each test method. One such method should be present before each test class. |
| [TestCleanup] | Marks a method that should be called after each test method. One such method should be present before each test class. |
| [TestClass] | Marks a class that contains tests. |
| [TestMethod] | Marks the method, i.e., the actual test case in the test class. |
| [DataRow] | Allows setting the values of the parameters of the tests. Multiple [DataRow] annotations can be present in the code. |
| [DataTestMethod] | It has the same functionality as the [TestMethod] attribute except that it is used when the [DataRow] attribute is used. |
| [AssemblyInitialize] | Marks the method that should be called once before the execution of any method in the assembly code. |
| [AssemblyCleanup] | Marks the method that should be called once after the execution of any method in the assembly code. |
| [Ignore] | Marks a test method or test class that should be considered for execution, i.e., it is ignored. |
| [TestCategory] | Specify the category for the test. |
| [ClassInitialize] | Methods that will be called only once before executing any of the test methods present in that class. |
| [ClassCleanup] | Methods that will be called only once after executing the test methods present in that class. |
MSTest V2 has cross-platform support and can be extended using custom test attributes & custom asserts. Parallel test execution is supported by the MSTest framework, where parallelism is possible at the Method level or Class level.
The test framework has evolved from being an “in-box” testing framework to a framework that is gaining wide adoption from the developer community.
This MSTest Tutorial for beginners and professionals will help you learn how to use MSTest framework with Selenium C# for performing Selenium automation testing.
NUnit vs. XUnit vs. MSTest
We now compare the C# unit testing frameworks from an attribute usage point of view along with a simple example, which demonstrates the code flow.
Attributes in test frameworks
Irrespective of the C# unit testing framework, attributes are used to describe class, methods, properties, etc. Attributes are meta-tags that provide additional information about the implementation under that particular tag.
To make the NUnit vs. XUnit vs. MSTest comparison clearer, we cover the important attributes in each of these frameworks. The NUnit vs. XUnit vs. MSTest attributes comparison will also help in porting the test implementation from one test framework to another.
| Marks a test method/individual test | [Test] | [TestMethod] | [Fact] |
| Indicates that a class has a group of unit tests | [TestFixture] | [TestClass] | N.A |
| Contains the initialization code, which is triggered before every test case | [SetUp] | [TestInitialize] | Constructor |
| Contains the cleanup code, which is triggered after every test case | [TearDown] | [TestCleanup] | IDisposable.Dispose |
| Contains method that is triggered once before test cases start | [OneTimeSetUp] | [ClassInitialize] | IClassFixture<T> |
| Contains method that is triggered once before test cases end | [OneTimeTearDown] | [ClassCleanup] | IClassFixture<T> |
| Contains per-collection fixture setup and teardown | N.A | N.A | ICollectionFixture<T> |
| Ignores a test case | [Ignore(“reason”)] | [Ignore] | [Fact(Skip=”reason”)] |
| Categorize test cases or classes | [Category()] | [TestCategory(“)] | [Trait(“Category”, “”) |
| Identifies a method that needs to be called before executing any test in test class/test fixture | [TestFixtureSetup] | [ClassInitialize] | N.A |
| Identifies a method that needs to be called after executing any test in test class/test fixture | [TestFixtureTearDown] | [ClassCleanUp] | N.A |
| Identifies a method that needs to be called before the execution of any tests in Test Assembly | N.A | [AssemblyInitialize] | N.A |
| Identifies a method that needs to be called after execution of tests in Test Assembly | N.A | [AssemblyCleanUp] | N.A |
To demonstrate the usage of attributes and the sequence in which the test code is executed, we take the example of simple test implementation.
NUnit Test Framework
In the example shown above, the implementation under [SetUp] and [TearDown] attributes will be called for each test case.
Shown below is the execution log:
|
1 2 3 4 5 6 7 |
Inside SetUp Inside TestMethod Test_1 Inside TearDown Inside SetUp Inside TestMethod Test_2 Inside TearDown |
xUnit Test Framework
As seen in the example shown below, the [SetUp] and [TearDown] attributes from the NUnit framework are not present. Instead, a constructor is used for initialization, and a Dispose method is used for de-initialization.
xUnit creates a new instance of the test class for every test. In some time-critical test scenarios, execution of creation & cleanup code may take time and make the tests slower. Class fixtures can be used in such situations where a fixture instance is created before any of the tests are executed, and once the tests have completed execution, cleanup of the fixture object by calling Dispose (if it is present).
In the example, we make use of Class Fixtures which means that even though there are two test classes, initialization, and cleanup code will be called only once.
Shown below is the output log:
|
1 2 3 4 5 |
Inside SetUp Constructor Inside Test_1 Inside Test_2 Inside CleanUp or Dispose method |
MSTest Test Framework
The test code only contains Console.WriteLine, which is put to trace the execution flow. Most of the widely used MSTest attributes are used in the demonstrated test code.
Shown below is the execution log:
|
1 2 3 4 5 6 7 |
Inside ClassInitialize Inside TestMethod Test_1 Inside AssemblyInitialize Inside TestInitialize Inside TestMethod Test_2 Inside TestCleanup |
The implementation under [ClassInitialize] & [ClassCleanup] attributes are called respectively once before the methods are executed and once after the execution of the test methods. The implementation under [TestInitialize] and [TestCleanup] attributes are respectively called once before & after executing tests in each class.
Take this certification to master the fundamentals of Selenium automation testing with C# and prove your credibility as a tester.
Here’s a short glimpse of the Selenium C# 101 certification from LambdaTest:
Core Differences between NUnit, XUnit, and MSTest
xUnit framework is different from NUnit and MSTest frameworks on multiple fronts. Let’s look at the core differences:
1. Isolation of Tests
xUnit framework provides much better isolation of tests in comparison to NUnit and MSTest frameworks. For each test case, the test class is instantiated, executed, and is discarded after the execution. This ensures that the tests can be executed in any order as there is reduced/no dependency between the tests. Executing each test as a separate instance minimizes the chances of one test causing the other tests to fail!
There is also an option to share setup/cleanup code that can be shared between different tests without the need to share object instances. One way of doing this is via Class Fixtures which we demonstrated earlier. xUnit has excellent documentation on how to share contexts between tests.
2. Extensibility
When we do NUnit vs. XUnit vs. MSTest, extensibility plays an important role in choosing a particular test framework. The choice might depend on the needs of the project, but in some scenarios, extensibility can turn the tables around for a particular test framework.
When compared to MSTest and NUnit frameworks, xUnit framework is more extensible since it makes use of [Fact] and [Theory] attributes. Many attributes that were present in NUnit framework e.g. [TestFixture], [TestFixtureSetup], [TestFixtureTearDown] [ClassCleanup], [ClassInitialize], [TestCleanup], etc. are not included in the xUnit framework.
Introduction of the [Theory] attribute for parameterized tests is one of the prime examples of the extensibility of the framework. This also makes the implementation of custom functionality much easier in the xUnit framework.
3. Initialization and De-initialization
The NUnit uses [SetUp], [TearDown] pairs whereas MSTest uses [TestInitialize], [TestCleanup] pairs for setting up the activities related to initialization & de-initialization of the test code.
On the other hand, xUnit uses the class constructor for the implementation of steps related to test initialization and IDisposable interface for the implementation of steps related to de-initialization.
xUnit starts a new instance per test, whereas, in NUnit & MSTest frameworks, all the tests execute in the same Fixture/Class.
4. Assertion mechanism
xUnit framework makes use of Assert.Throws instead of [ExpectedException] which is used in NUnit and MSTest. The drawback of using [ExpectedException] is that the errors might not be reported if they occur in the wrong part of the code. For example, if assert has to be raised for Security Exception, but Authentication Exception occurs, [ExpectedCondition] will not raise assert.
On the other hand, Assert.Throws raises assert even if the exception is generic. This ensures that assert is raised even after the exception is raised.
5. Parallel test execution
All the three C# unit testing frameworks support parallel test execution and are well-suited for Selenium automation testing as throughput plays a major role in automation testing. Below are the ways in which parallelism can be achieved in each of the test frameworks.
- NUnit – Parallelism is possible at the level of Children (child tests are executed in parallel with other tests), Fixtures (descendants of test till the level of Test Fixtures can execute in parallel), Self (test itself can be executed in parallel with other tests), and All (test & its descendants can execute in parallel with others at the same level).
1234[Parallelizable(ParallelScope.Children)][Parallelizable(ParallelScope.Fixtures)][Parallelizable(ParallelScope.Self)][Parallelizable(ParallelScope.All)]
- xUnit – Parallelism by putting the test classes into a single test collection or by executing an ‘n’ number of threads in parallel.
- MSTest – Parallelism at Method level as well as Class level.
123[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)][assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]
|
1 2 3 |
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] [assembly: CollectionBehavior(MaxParallelThreads = n)] |
Note: Workers indicate the number of threads that can be executed in parallel.
NUnit vs. XUnit vs. MSTest – Demonstration using Remote Selenium Grid
For the initial rounds of cross browser testing can be performed on the local Selenium as it aids in verifying the functionalities of the test code on ‘certain’ combinations of browsers & operating systems. Selenium automation testing can be performed using either of the three C# unit testing frameworks.
Automation testing using the local Selenium Grid is a good start to cross browser testing activity, but local testing may not be sufficient to attain good test coverage. Having in-house infrastructure with machines having different web browsers and operating systems installed is not a scalable & economic idea.
When cross browser testing is performed using local Selenium WebDriver, the activity can hit a roadblock as you cannot perform Selenium automation testing using innumerable combinations of web browsers, operating systems, and devices. Instead of the local Selenium Grid, the capabilities of the remote Selenium Grid on the cloud can be leveraged so that testing is more fool-proof.
LambdaTest is an AI-powered test orchestration and execution platform where you can test websites and mobile applications on 3000+ combinations of browsers, devices, and operating systems. It seamlessly integrates with popular unit testing frameworks like NUnit, xUnit, and MSTest. This means you can continue writing your unit tests using your preferred framework while harnessing the power of LambdaTest for comprehensive cross-browser and cross-platform testing.
With LambdaTest, you can effortlessly execute your unit tests across a wide range of browser and OS combinations, ensuring your code is robust and reliable for all users, regardless of their choice of browser or device. The effort to port the code from the local Selenium Grid to LambdaTest’s remote Selenium Grid is not much as major changes are infrastructure-related.
After creating an account on LambdaTest, make a note of the user-name and access-key from the Profile Section. The Dashboard consists of all the relevant information about the tests, status, logs, video recordings of the tests, etc. LambdaTest capabilities generator is used to generate the desired browser capabilities against which the cross browser tests will be performed. Shown below are the capabilities for Safari on the macOS Mojave platform:
|
1 2 3 4 5 6 7 8 |
DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.SetCapability("user","Your Lambda Username") capabilities.SetCapability("accessKey","Your Lambda Access Key") capabilities.SetCapability("build", "your build name"); capabilities.SetCapability("name", "your test name"); capabilities.SetCapability("platform", "MacOS Mojave"); capabilities.SetCapability("browserName", "Safari"); capabilities.SetCapability("version","12.0"); |
RUN YOUR CSHARP SCRIPT ON SELENIUM GRID
3000+ Browsers AND OS
Implementation (using NUnit C# unit testing framework)
To demonstrate the usage of Remote Selenium WebDriver on LambdaTest Grid, we implement the below test case:
- Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using selected browsers.
- Check the first two items.
- Add a new item – Adding item to the list.
- Click the Add button to add that new item to the list.
Below are the browser and operating system combinations on which the test has to be performed.
The browser capabilities are generated using the LambdaTest capabilities generator. The combination of user-name & access-key is used to access LambdaTest’s remote Selenium Grid.
Parallel test execution is enabled using the Parallelizable attribute, and test combinations are passed via TestFixtures. The actual logic of the test code is under the [Test] attribute. Shown below is the snapshot from the Automation Tab on LambdaTest and Visual Studio:
NUnit Test Automation Using Selenium C# (with Example)
Implementation (using xUnit C# unit testing framework)
For demonstrating parallel test execution using xUnit framework on remote Selenium Grid, we test the following scenarios:
Test Case 1 – LamdaTest ToDo App
- Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using the Firefox WebDriver.
- Mark the first two items as Done, i.e., Check those two items.
- Add a new item – Adding an item to the list.
- Click the Add button to add that new item to the list.
Browsers on which cross browser testing is performed are:
Test Case 1
Test Case 2 & 3 – Google Search for LambdaTest
- Navigate to Google.com.
- Search for LambdaTest.
- Quit the browser window.
Both the test cases are the same, but the execution will be performed on different web browsers.
Test Case 2
Test Case 3
The setup/initialization code is a part of the constructor, whereas the implementation under IDisposable [i.e., public void Dispose()] can be used to perform the de-initialization. For simplification, de-initialization is moved to the individual test cases. Shown below is the output snapshot from LambdaTest and Visual Studio.
Implementation (using MSTest C# unit testing framework)
To demonstrate the usage of parallelism with the MSTest framework, we implement the test cases used for the xUnit framework. The browser and operating system combinations for testing are also the same.
Each test case is under a separate [TestMethod] attribute. Since we have implemented three tests (LT_ToDo_Test, Google_Test_1, and Google_Test_2), three concurrent sessions will be executed till the time the number of pending tests are less than three. The [DataRow] attribute is used to create parameterized tests.
Shown below is the output snapshot from the Automation tab on LambdaTest and Visual Studio:
MSTest Tutorial: Running First Selenium Automation Script
Conclusion
Each of the .NET frameworks that we have discussed so far has evolved over a period of time. All these frameworks are well-suited for cross browser testing as they have support for Selenium. MSTest V2 is a giant leap in terms of usability when compared to MSTest V1. The xUnit framework is more sophisticated as it is more community-focused, and the creators had immense learning while building the NUnit framework.
To conclude, the NUnit vs. XUnit vs. MSTest comparison is more about which C# test framework is well-suited to perform your job and the test expertise that your team possesses. If I had an option to select a framework, I would go with xUnit since it is more extensible and has fewer attributes, making the code clean & easy to maintain.
What is your favorite C# test framework, do leave your choice & relevant explanation in the comments section.
Frequently Asked Questions
Is xUnit better than MsTest?
xUnit is, by default, leaner and cleaner than both NUnit and MSTest when it comes to writing unit tests. This feature is one of many that enables us to help write cleaner tests.
Which is better NUnit or MsTest?
The main difference between MsTest and NUnit is the ability to execute tests in parallel at the method level. MsTest is a framework through which developers can run unit tests for .NET applications. It provides advantages in both speed and robustness when compared to NUnit.
Should I use NUnit or xUnit?
xUnit was created to succeed NUnit, a popular unit testing library that is part of the .NET framework. Although the .NET framework has evolved since NUnit was first written, xUnit leverages some of its advanced features to write cleaner tests that are easier to debug and run than in NUnit.
Author’s Profile
Himanshu Sheth
Himanshu Sheth is a seasoned technologist and blogger with more than 15+ years of diverse working experience. He currently works as the 'Lead Developer Evangelist' and 'Senior Manager [Technical Content Marketing]' at LambdaTest. He is very active with the startup community in Bengaluru (and down South) and loves interacting with passionate founders on his personal blog (which he has been maintaining since last 15+ years).
Blogs: 132
Got Questions? Drop them on LambdaTest Community. Visit now
