microsoft / testfx

MSTest framework and adapter

Microsoft Test Framework

MSTest, Microsoft Testing Framework, is a unit testing framework for .NET applications. It allows you to write tests, use Test Explorer, create test suites, and apply the red, green, and refactor patterns to write code.

This is a fully supported, open source, and cross-platform test framework with which to write tests targeting .NET Framework, .NET Core, .NET, UWP, and WinUI on Windows, Linux, and Mac.

How can I contribute?

We welcome any kind of contribution!

• Contributing guides how to best contribute
• Dev Guide explains how to build and test
• Documentation contains information about the history, context, and supported or unsupported features. It also gathers the various official documentation pages on learn.microsoft.com about MSTest.

How to consume MSTest?

MSTest is shipped as NuGet packages that can be added to your projects. The following table lists all available packages.

Name	Description	Stable version	Preview version	Dogfood version
MSTest	This package is a meta package that simplifies referencing all recommended MSTest packages.			Azure Artifacts
MSTest.TestFramework	This package includes the libraries for writing tests with MSTest. To ensure the discovery and execution of your tests, install the `MSTest.TestAdapter`` package.			Azure Artifacts
MSTest.TestAdapter	This package includes the adapter logic to discover and run tests. For access to the testing framework, install the MSTest.TestFramework package.			Azure Artifacts
MSTest.Analyzers	This package includes code analyzers and code fixes for MSTest.			Azure Artifacts

License

MSTest is licensed under the MIT license.

The LICENSE and ThirdPartyNotices in any downloaded archives are authoritative.

testfx's Issues

Multiple copies added for same test on running multiple times in IntelliTest

Description

Running IntelliTest multiple times, results in multiple copies of the same test being added to the test class with an incremental suffix in the name. This is an issue with MSTestV2 and works for MSTest.

Steps to reproduce

1. Create a Class Library.
2. Right CLick -> IntelliTest -> Create IntelliTest with MSTestV2 framework
3. RightClick-> IntelliTest -> Run IntelliTest multiple times

Expected behavior

*.Test.g.cs should have one copy of all possible test cases.

Actual behavior

*.Test.g.cs have multiple copies of the same test being added to the test class with an incremental suffix in the name

Environment

VS2017 RC4

Upgraded to MSTest v2, The type initializer for 'Microsoft.VisualStudio.TestTools.UnitTesting.TestConfiguration' threw an exception

Question

Has anything changed around the DataSource attribute in MSTest v2? Were there any upgrade steps I need to perform?

Description

Have upgraded to MS Test v2 (v 1.1.13). Running my unit test gives:

Message: The type initializer for 'Microsoft.VisualStudio.TestTools.UnitTesting.TestConfiguration' threw an exception.

Result StackTrace:	
at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDataSource.GetConnectionProperties(DataSourceAttribute dataSourceAttribute, String& providerNameInvariant, String& connectionString, String& tableName, DataAccessMethod& dataAccessMethod)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDataSource.RunDataDrivenTest(TestContext testContext, ITestMethod testMethodInfo, ITestMethod testMethod, TestMethodAttribute executor)

Steps to reproduce

I have the following attributes on my test:

    [TestMethod]
    [DataSource("NoteData")]
    [DeploymentItem(@".\TestFiles\ExcelFiles\TestData.xls")]
    [DeploymentItem(@".\TestFiles\Note", "NoteAck")]
    [DeploymentItem(@".\TestFiles\Note", @"TestFiles\NoteAck")]

In my app.config I have

 <microsoft.visualstudio.testtools>
    <dataSources>
      <add name="NoteData" connectionString="TestDataConnection" dataTableName="IssueNoteAck$" dataAccessMethod="Sequential" />
    </dataSources>
  </microsoft.visualstudio.testtools>

<connectionStrings>    
    <add name="TestDataConnection" connectionString="Driver={Microsoft Excel Driver (*.xls)}; dbq=TestData.xls; defaultdir=.; driverid=790; maxbuffersize=2048; pagetimeout=5; readonly=true;" providerName="System.Data.Odbc" />
  </connectionStrings>

Expected behavior

Test would pass as previously

Actual behavior

As above with exception and stack trace

Environment

Using VS 2017 15.0.0-RTW+26228.4
Win 7
  <package id="MSTest.TestAdapter" version="1.1.13" targetFramework="net462" />
  <package id="MSTest.TestFramework" version="1.1.13" targetFramework="net462" />
.Net Framework v4.6.2

Ability to provide a test timeout for every test in a session.

Description

Currently the timeout for a test is only an attribute apply-able on a test method in code. This cannot be changed without a rebuild of the test code. This request is to add a configurable timeout for each test in a session. This can be achieved via an additional setting in the runsettings section for the adapter like below:

<Runsettings>
   <MSTest>
      <TestTimeout>2000</TestTimeout>
   </MSTest>
</Runsettings>

So when the above runsettings is provided in a session, every test in that session can run for a maximum of 2000 milliseconds.

Problems with null in DataRow

Copied from microsoft/vstest#471 on behalf of @jmagaram

Description

Occasionally my unit tests fail to run and display a very cryptic message in the output. I've narrowed the problem down to using null values inside a DataRow. The error message is completely unhelpful and the fix is unexpected. I can't believe this is by-design.

[DataTestMethod]
[DataRow(false, "abc")]
[DataRow(true, "")]
[DataRow(true, "   ")]
[DataRow(true, null)]
public void IsStringNullOrWhitespace(bool expectedResult, string s)
{
    bool result = String.IsNullOrWhiteSpace(s);
    Assert.AreEqual(expectedResult, result);
}

The error message is Error in executing test. No result returned by extension. If using extension of TestMethodAttribute then please contact vendor.

The "fix" is to replace the last row with...

[DataRow(true, (string)null)]

When I do this, Visual Studio recommends removing the unnecessary cast. I've tried labeling the datarow parameters but can't seem to make this work.

Steps to reproduce

Try running the unit test above in the RC version of Visual Studio 2017.

Expected behavior

I expect the test to run and give me correct results. Or a much more helpful error message.

Templates and Wizards vsix should be built with RC3 tooling

Description

The templates and wizards tooling currently is built with an RC or older tooling based on the build machines configured in the lab. This could cause issues with out-of-band upgrade. This needs to be dependent on RC3 tooling instead.

Upgrade to RTM version of nugets in .Net Core templates

Move the nuget packages for MSTest V2 to 1.0.8-rc2 and 1.1.10-rc2 for .Net core projects by default.

MSTest: Constructor runs for every test (rather than once for the class) during Debug

Copied from dotnet/project-system#1192

If you Debug an MSTest with one or more test methods, the constructor runs once for every test (rather than once for the entire class). This does not seem to occur when running (not debugging). The following code demonstrates the issue - failing when debugging (successful when not debugging):

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace UnitTestProject1
{
[TestClass]
public class TestBase
{
}

[TestClass]
public class UnitTestSample
{
    static public int ConstructorCount { get; set; }

    public UnitTestSample()
    {
        ConstructorCount++;
    }

    [TestMethod]
    public void TestMethod1()
    {
    }

    [TestMethod]
    public void TestMethod2()
    {
    }

    [ClassCleanup]
    static public void ClassCleanup()
    {
        Assert.AreEqual<int>(1, ConstructorCount);
    }
}
}

DeploymentItem should be implemented via Fx extensibility

Description

Deployment Item currently is hard-coded into the adapter's core platform service leading to a bunch of dependency issues with discovery in projects targeting UWP. This should truly be separated out as a separate module so that:

1. Deployment as a pillar is extensible to anyone wanting to have their own custom deployment.
2. The design as such is clean, since this is specific only to certain platforms.

ExpectedException Support

Description

Add back support for Expected Exception which was made obsolete. This would helps in two ways:

1. Transition from V1 would be made much simpler.
2. The Create IntelliTest Wizard which currently only generates method level attributes would continue to work until fixed later on.

Steps to reproduce

1. Create a Unit Test project.
2. Add the V2 test fx and adapter to it.
3. Add a test method which throws an exception and add an ExpectedException attribute targeting the same exception.

Expected behavior

The test should pass.

Actual behavior

The test fails and the build throws a warning that this is an obsolete attribute.

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

Extensibility: data driven testing

Description

Provide the ability to extend the data source to your favorite store, generate data, use combinatorics or pairwise testing.

Make Exceptions Serializable

Description

Currently AssertFailedException, AssertInconclusiveException, InternalTestFailureException, and UnitTestException are not marked serializable.

This presents a problem when executing tests in an different AppDomain. Since the exceptions are not serializable the runtime can't move them between app domains and throws SerializationException.

AB#1575813

Deprecate ExpectedException attribute.

Description

I can still write tests that use ExpectedExceptionAttribute.

Steps to reproduce

Write this test
[TestMethod]
[ExpectedException(typeof(ArgumentException))]
public void TestMyStuff()
{
// arrange
var instanceUnderTest = new YourFunkyClass();
DoSomethingThatThrowsArgumentException();

// act // assert
Assert.ThrowsException(()=>instanceUnderTest.DoStuff("badargument");}
}

Expected behavior

I understand backwards compatibility, but ExpectedException is WIDELY recognized to be a source of tests passing when they should be failing. Any test with ExpectedException is effectively useless unless there are absolutely no other calls in the test. You just don't know if it's failing for the reason you expect it to fail, or if it's even getting to the line you want it to fail on.

At a minimum, there needs to be a way to set the test runner to reject it so that if someone brings it in, it fails. By MSTest 3, the attribute should go away entirely.

Actual behavior

I can still use it.

Environment

plain Win10, VS2017.

AB#1934068

CollectSourceInformation awareness to query source information

Description

Per this issue the adapter is now aware if it is driven by an IDE/CLI. Querying source information for a test method is only required in IDE scenarios where one can run a test from an editor context for instance. Getting source information from the Test Platforms DIA wrapper is a costly operation and is of no use in CLI scenarios.
Filing this issue to conditionally not query for this information when running from CLI to better performance.

Environment

MSTest.TestFramework: 1.1.13
MSTest.TestAdapter: 1.1.13

Extensibility: Custom execution

Description

Control test execution in various ways:

1. constraint on the execution time, memory/CPU usage, of a test
2. constraining the application platforms on which the test can execute, etc.

Packaging mstest for .net core

Description

In .net core, the dependencies are resolved from deps.json. The adapter must be referenced from a test project such that the MSTest.TestAdapter.dll is located by the test platform.

Currently, the nuget package ships with props files which copy the test adapter to output directories. It is not supported in .net core (see #6). What is the way forward?

• Ship different nuget package for each runtime
• Ship single nuget package, let the adapter be resolved from the corresponding runtime directory e.g. netcoreapp1.0 for .net core.

Thoughts?

[Test] Desktop CLI Tests do not verify stack trace information.

Description

Desktop CLI Tests do not show stacktrace information correctly for x64 scenarios and hence cannot be validated.

Steps to reproduce

The code at test/E2ETests/Automation.CLI/CLITestBase.ValidateFailedTests() skips verification of stack trace information in x64. This needs to be resolved.

Extensibility: Action attributes

Description

Allow all test lifecycle attributes to be extensible:

• TestInitializeAttribute
• ClassInitializeAttribute
• AssemblyInitializeAttribute
• TestCleanupAttribute
• ClassCleanupAttribute
• AssemblyCleanupAttribute

Also allow multiple action attributes on a test method.

Failures in Assembly and Class Cleanup do not fail the Tests in them.

Description

Any exceptions/failures in the assembly/class cleanup methods do not fail the containing test methods. It instead displays a warning with exception details.

Steps to reproduce

1. Create a Test project and add the following class:

[TestClass]
    public class UnitTestSample2
    {
        static public int ConstructorCount { get; set; }

        public UnitTestSample2()
        {
            ConstructorCount++;
        }

        [TestMethod]
        public void TestMethod1()
        {
        }

        [TestMethod]
        public void TestMethod2()
        {
        }

        [ClassCleanup]
        static public void ClassCleanup()
        {
            Assert.AreEqual<int>(1, ConstructorCount);
        }
    }

2. Run Tests from VS IDE

Expected behavior

The tests fail.

Actual behavior

The test pass with the following warning in the Output -> Tests pane:
"Class Cleanup method UnitTestSample.ClassCleanup failed. Error Message: Assert.AreEqual failed. Expected:<1>. Actual:<2>. . Stack Trace: at UnitTestProject15.UnitTestSample.ClassCleanup() in c:\users\aajohn\documents\visual studio 2017\Projects\CoreUnitTestProject15\CoreUnitTestProject15\UnitTest1.cs:line 29"

Environment

Framework and Adapter version : 1.1.11

Note

This behavior is similar to the legacy framework. Filing this to ensure we have the right experience.

WorkItem Attribute needs to be added in.

Description

WorkItemAttribute was in V1 and migrating to V2 causes build failures. This needs to go via an extensibility point though.

Steps to reproduce

1. Create a MSTest V1 project with a WorkItemAttribute.
2. Migrate to MSTest V2.
3. Build project

Expected behavior

Project build works.

Actual behavior

Build fails with a cannot find a WorkItemAttribute.

Environment

This should repro across.

Build warning on reference to MSTest.TestFramework

Issue type

Please specify the type of issue:

• Specification
• Code defect
• Question

Steps to reproduce

• Create a new project.json based test project that targets both net46 and netcoreapp1.0
• dotnet build the project

It reproduces on running build.cmd in vstest repository.

Expected behavior

No warnings.

Actual behavior

A testproject.nuget.targets is created with following text:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="EmitMSBuildWarning" BeforeTargets="Build">
    <Warning Text="Packages containing MSBuild targets and props files cannot be fully installed in projects targeting multiple frameworks. The MSBuild targets and props files have been ignored." />
  </Target>
</Project>

Environment

• dotnet --version
  1.0.0-preview3-003546

MSTestv2 does not support testSettings. Are there features that need to be brought fwd to runSettings?

Description

TestSettings are not supported in MSTestv2. Are there features that need to brought forward in runsettings?

An inconclusive in a test initialize fails the test if it has an ExpectedException

Description

If a test is reported inconclusive in its TestInitialize but is adorned with a ExpectedException attribute, it fails.

Steps to reproduce

1. Create a test project with the following code:

        [TestInitialize]
        public void TestInit()
        {
            Assert.Inconclusive();
        }

        [TestMethod]
        [ExpectedException(typeof(NullReferenceException))]
        public void TestMethod1()
        {
        }

2. Run the test.

Expected behavior

The test is inconclusive.

Actual behavior

The test fails.

Environment

MSTest.TestFramework: 1.1.14.
MSTest.TestAdapter: 1.1.14.

Run tests fail intermittently with a disconnected from server exception.

Description

Running a large set of tests sometimes throws the following exception from the Test Platform:
"Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Object '/ddbf5695_3650_4819_8ba2_77ea8202b458/ec2ln5fvtuhgu_udubuqgkpz_27.rem' has been disconnected or does not exist at the server. "

Steps to reproduce:

This is observed when running a large set of tests on a machine with lower config.

Expected behavior

The tests should all run.

Actual behavior

They intermittently fail mid-way.

TestMethod failures are masked by TestCleanup exceptions in the outcome.

Description

Failures in the TestMethod are masked by TestCleanup exceptions in the outcome.

Steps to reproduce

1. Create an MSTest V2 based unit test project.
2. Throw an exception in the TestMethod and the TestCleanup.
3. Run the test.

Expected behavior

An indication of the test method exception and stack trace should be shown in the outcome.

Actual behavior

Only the TestCleanup stack is shown.

Support DeploymentItem in .Net Core projects

Description

[DeploymentItem] attribute is a no-op in the .Net Core world. This is however supported in Desktop and is required for functional testing scenarios and to provide isolation for test runs. Filing this work-item to add in that support for .Net Core based projects as well.

Steps to reproduce

1. In a .Net Core project add a DeploymentItem on a class/method linking to an xml resource(say) needed for the run.
2. Run a test that reads this xml resource.

Expected behavior

The test deploys the xml resource to a deployment directory and runs the test from there.

Actual behavior

The test fails complaining about not being able to find the resource file.

Implement UITestMethod in-box

Description

-UWP tests built on V1 that have a UITestMethod do not build after switching to V2 since we have not ported this attribute over to the V2 framework.

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, sample project, target platform (desktop, core, uap)

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

Description Attribute needs to be added in.

Description

DescriptionAttribute was in V1 and migrating to V2 causes build failures. This needs to go via an extensibility point though.

Steps to reproduce

1. Create a MSTest V1 project with a DescriptionAttribute
2. Migrate to MSTest V2.
3. Build project

Expected behavior

Project build works.

Actual behavior

Build fails with a cannot find a DescriptionAttribute

Environment

This should repro across.

TestContext.Writeline does not output messages.

Description

This came from a user report on a blog.
TestContext.WriteLine does not output the messages anywhere.

Steps to reproduce

In a vanilla desktop unit test project, add the following:

public TestContext TestContext
{
    get;
    set;
}

[TestMethod]
public void TestMethod1()
{
     this.TestContext.WriteLine("Hello There");
}

Run the test from Visual Studio.

Expected behavior

That "Hello There" is printed in the TestOutput pane.

Actual behavior

Nothing is printed anywhere.

Environment

MSTest.TestFramework : 1.1.13
MSTest.TestAdapter : 1.1.13

Ability to provide a reason for Ignored tests

Description

Tests tagged with the IgnoreAttribute do not have the ability to state a reason why these tests are ignored. This requires users to search through the test code to figure out why it was disabled. The ask here is to have something similar to

[Ignore("Enable after refactoring.")]

This reason would then start showing when test results are displayed in trx/VS IDE/CLI

Here is the uservoice item that requests this ability.

Tests with Deployment Item do not run.

Description

A test with a DeploymentItem attribute does not work.

Steps to reproduce

1. Add a "DeploymentItem" on a test method with a resource file to deploy.
2. Run the test.

Expected behavior

The test should drop all the binaries to the deployment directory and run tests.

Actual behavior

The tests do not run with the following message in the "Tests" tab:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

Environment

VS IDE. Works fine from commandline.
MSTest.TestFramework 1.1.11
MSTest.TestAdapter 1.1.12

Extensibility: Custom assertions

Description

Enable users to define custom assertions that plug-in to the test framework’s assertion infrastructure.
RFC: Framework Extensibility for Custom Assertions

Add support for ordered tests

Description

Is it possible to decide this based on telemetry data of ordered tests usage?

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, sample project, target platform (desktop, core, uap)

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

[UWP] [loc] in release does not localize messages from TestPlatform Framework

Description

Messages from Framework are not localized

Steps to reproduce

1. Install VS on a German machine, & create a UWP Test project
2. Add Assert.AreEqual(1,2)
3. Compile in release mode, & run tests.

Expected behavior

Failure message should be localized

Actual behavior

Failure message is not localized

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

VB unit test project templates that ship in VS 2017 do not reference MSTest V2 nuget packages

From @shyamnamboodiripad, microsoft/vstest#673

Description

Live Unit testing does not work out of the box for VB unit test projects created using VS 2017

Steps to reproduce

1. Create a new VB Unit Test Project in VS 2017 -
   
2. Build the project
3. Turn on LUT (Test -> Live Unit Testing -> Start)

Expected behavior

LUT is turned on

Actual behavior

[14:16:35.202 Error] No test adapters are referenced by this solution. If you have a test project, add a NuGet reference to a test adapter corresponding to the test framework used in order to run the tests. If you already have the required NuGet reference in your test project, performing a NuGet restore may resolve the issue.
[14:16:35.202 Info] Start failed.
[14:16:35.264 Info] Live Unit Testing stopped.

Looks like VB project template for Unit Test Project is still adding a reference to the old 'MS.VS.QualityTools.UnitTestFramework.dll' instead of 'MS.VS.TestPlatform.TestFramework' nuget packages. Corresponding C# project template references the correct version.

Referencing the same nuget packages in the created VB project fixes the issue.

Environment

VS 2017 Enterprise

TestContext.WriteLine throwing System.FormatException when json string is passed.

Description

TestContext.WriteLine throwing System.FormatException when json string is passed.

Method Signature:
public abstract void WriteLine(string format, params object[] args);

Steps to reproduce

• Create a unit test project, add MSTest.TestFramework dll and execute the below test method

[TestClass]
    public class UnitTest1
    {
        private TestContext testContextInstance;
        /// <summary>
        ///Gets or sets the test context which provides
        ///information about and functionality for the current test run.
        ///</summary>
        public TestContext TestContext
        {
            get
            {
                return testContextInstance;
            }
            set
            {
                testContextInstance = value;
            }
        }
        [TestMethod]
        public void TestMethod1()
        {
            var msg = "{curly braces}";
            //Debug.WriteLine(msg);
            TestContext.WriteLine(msg);
        }
    }

Expected behavior

Test sholuld pass, user should be able to write the json in output.

Actual behavior

Test is failing with System.FormatException.

RCA :
This is happening because string.format is used for implementing this API and it expects format which placeholders {0}, {1}, etc. and corresponding arguments.

But when json string is passed, it just throws System.FormatException as string is present inside curly braces.

Workaround:
As a workaround, user has to add one extra curly brace for every curly brace present in string.

string str = "{{mystring}}";

Fix :
Do not use string.format api when count of arguments are null in implementation of API
public abstract void WriteLine(string format, params object[] args);

Core platform service built for the latest Core CLR

Description

The core platform service is currently built for net standard 1.0 because of which a bunch of functionality be it Console Redirection or Code navigation is not possible. This needs to change to either build against netcoreapp1.0 or a later netstandard.

Steps to reproduce

N.A

Expected behavior

The platform service for core should be able to have access to a larger API set providing richer functionality for .Net Core.

Actual behavior

The platform service for .Net Core is built against netstandard1.0.

Environment

N.A

Inheritance support for base classes that resides in different assemblies.

Description

Try to have derive test class in different assembly than base test class.

Steps to reproduce

Add a base test class TestClassBase in assembly1.dll
Add a derived test class TestClassDerived in assembly2.dll

Expected behavior

Test methods in TestClassDerived class should be discovered/run

Actual behavior

Test methods on TestClassDerived.cs class not discovered/run

Environment

Package version of MSTest - preview-1.1.4

Exceptions in trace logging in the default workflow

Description

Exceptions related to Object Model resolution are seen in the default workflow in the Test Platform trace logs.

Steps to reproduce

1.Create a netcore project with MStestv2.
2. Build the project
3. Run rest from above project with parameter /diag:log.txt

Expected behavior

There should not be any false-positive exceptions in log.txt

Actual behavior

The log.txt has a couple of exceptions related to not being able to find Microsoft.VisualStudio.TestPlatform.ObjectModel with Version=11.0.0.0

Environment

Visual Studio 2015/2017
log.txt

Warning when vsmdi files are passed in.

Description

The adapter currently throws the following warning when a vsmdi file is passed to it:
"A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter."
This does not indicate that this is an unsupported scenario.

Steps to reproduce

1. unzip the mstestv2 project from the attachments.
2. Discover tests.
3. Ensure that the test settings via the Test-> TestSettings is pointing to a vsmdi file.
4. Discover Tests

Expected behavior

A warning that indicates this is an unsupported behavior.

Actual behavior

The following warning shows up: "A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter."

Repro.zip

Console.Writeline in a TestMethod are not available in test output for .net core

Description

Console.WriteLine in a test method should be available in output.

Steps to reproduce

1. Create a new test project
2. Add a test method
3. Add Console.WriteLine("foo") in test method

Expected behavior

A output link should appear in test explorer summary

Actual behavior

Output link is not shown.

Environment

VS: WPT/25810.00
VSTest: as of ce0eac2

Copied from microsoft/vstest#132

Deployment directories should be at a per source level.

Description

Separate Deployment directories should be created for each source that requires deployment item

Steps to reproduce

1. Create two C# UnitTest Projects.
2. Add Deployment Item in only one of the projects and not in second one.
3. Build the solution.
4. Put breakpoints in both the tests.
5. Select both the tests in Test Explorer and do debug selected tests.

Expected behavior

Only one test dll having [DeploymentItem] attribute should have been copied to Deployment Directories(i.e. UnitTestProject76) and other one should have run from UnitTestProject1/bin/debug and should not get copied to Deployment Directory.

Actual behavior

Both the test dlls gets copied in the Deployment Directories that are created.

This actually changes the behavior of tests when run on their own versus when run with other test assemblies that have deployment items causing confusion.

Source info not displaying in Test details window

Description

Source info not displaying for netcore projects. Diasession now available in for netcore.

Steps to reproduce

Create netcore unit test project from template.
Select a test in test explorer.

Expected behavior

Source Info should display.

Actual behavior

source info in test details window is not displaying.

Environment

Added DiaSession support to ObjectModel in fix for microsoft/vstest#46

UITestMethods don't appear in Test Explorer

Description

In a UWP test project, methods attributed with UITestMethod don't appear in the Test Explorer. They reappear if the attribute is changed to TestMethod.

Steps to reproduce

1. Create a new UWP test project.
2. Build. Notice that TestMethod1 shows up in the Test Explorer.
3. Change [TestMethod] to [UITestMethod] on TestMethod1.
4. Build.

Expected behavior

TestMethod1 still appears in Test Explorer.

Actual behavior

It disappears.

Environment

I am using RC4 with the latest prerelease MSTest framework and adapter.

Extensiblity: Custom attributes

Description

Enable users to define strongly typed description attributes as opposed to <string, string> key value pairs.
RFC: Framework Extensibility Description Attributes

Ability to add custom assertions.

Description

Often times, the default set of assertion APIs are not sufficient to satisfy a wide range of requirements for unit test writers. In most of these situations users end up having utility methods to address this need reducing discoverability in a test suite. If the test framework provides an extensibility in the assertion infrastructure itself, custom assertion functionality can be

• Easily accessed
• Easily organized and
• Possibly shared with the community.

Filing this issue to plug in an extensibility model to address this.

Show warning/info at compile time when [DeploymentItem] attribute used for .NetCore or UWP project

Description

Currently user have no way to figure out that [DeploymentItem] is not supported for UWP/.Net Core projects. Adding a meaningful warning/information message at compile time can save both of our time and developer's time.

Steps to reproduce

1. Create a UWP test project.
2. Add [DeploymentItem] attribute on top of a testmethod
3. Build and run tests

Expected behavior

Mentioned item should get copied over to deployment directory

Actual behavior

No item is copied.

Consider adding ApartmentState as extension attribute

Description

Enable to run a test method with apartment state as STA/MTA.

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, sample project, target platform (desktop, core, uap)

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

TestContext.CurrentTestOutcome is always InProgress in the TestCleanup method

Copied from #Microsft/vstest/548.

Description

In the TestCleanup method, the TestContext.CurrentTestOutcome property is always InProgress rather than the actual current test outcome.

Using the old test framework (bundled with VS, New > Project > Unit Test) the outcome is set correctly.

Steps to reproduce

using Microsoft.VisualStudio.TestTools.UnitTesting;

[TestClass]
public class MyTestClass
{
	public TestContext TestContext { get; set; }

	[TestMethod]
	public void PassTest()
	{
		Assert.IsTrue(true);
	}

	[TestMethod]
	public void InconclusiveTest()
	{
		Assert.Inconclusive();
	}

	[TestMethod]
	public void FailTest()
	{
		Assert.Fail();
	}

	[TestCleanup]
	public void TestCleanup()
	{
		System.Console.WriteLine($"{TestContext.TestName}: {TestContext.CurrentTestOutcome}");
	}
}

Expected behavior

TestContext.CurrentTestOutcome is set to the actual outcome type of the current test.

Image taken when executing the code above using the old VS-bundled framework.

Actual behavior

TestContext.CurrentTestOutcome is always InProgress

Image taken when executing the code above using the new NuGet package framework.

Environment

	Version
OS	Windows 10 Enterprise build 14393
IDE	Visual Studio Enterprise 2015 Update 3
Target framework	4.5.2
MSTest.TestAdapter	1.1.12
MSTest.TestFramework	1.1.11

Error installing MSTest.TestFramework on .net core

Description

MSTest.TestFramework should support netcoreapp1.0 target.

Steps to reproduce

• VS from WPT, 25715.03
• Create a new console application
• Install MSTest.TestFramework

Expected behavior

Test framework installs correctly.

Actual behavior

Test framework install fails with following error:

Package MSTest.TestFramework 1.0.1-preview is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package MSTest.TestFramework 1.0.1-preview supports:
  - dotnet (.NETPlatform,Version=v5.0)
  - net451 (.NETFramework,Version=v4.5.1)
One or more packages are incompatible with .NETCoreApp,Version=v1.0.

Environment

MSTest.TestFramework 1.0.1-preview.

Logger.LogMessage logs a message mutliple times

Steps to reproduce

This came in from a user on a blog post. Add the following piece of code to a TestMethod and run it.

Logger.LogMessage("Hello World");

Expected behavior

Logs "Hello World" to the standard output stream.

Actual behavior

"Hello World" is logged twice to the Standard output and error streams!!

Environment

Scenario: VS IDE.
MSTest.TestFramework: 1.1.11
MSTest.TestAdapter: 1.1.11