dotnet / xdt

XDT is Microsoft's Xml Document Transformation library.

dotnet/dotnet - Home of the .NET VMR

This repository is a Virtual Monolithic Repository (VMR) which includes all the source code and infrastructure needed to build the .NET SDK.

What this means:

• Monolithic - a join of multiple repositories that make up the whole product, such as dotnet/runtime or dotnet/sdk.
• Virtual - a mirror (not replacement) of product repos where sources from those repositories are synchronized into.

In the VMR, you can find:

• source files of each product repository which are mirrored inside of their respective directories under src/,
• tooling that enables building the whole .NET product from source on Linux platforms,
• small customizations, in the form of patches, applied on top of the original code to make the build possible,
• [in future] E2E tests for the whole .NET product.

Just like the development repositories, the VMR will have a release branch for every feature band (e.g. release/8.0.1xx). Similarly, VMR's main branch will follow main branches of product repositories (see Synchronization Based on Declared Dependencies).

More in-depth documentation about the VMR can be found in VMR Design And Operation. See also dotnet/source-build for more information about our whole-product source-build.

Goals

• The main purpose of the dotnet/dotnet repository is to have all source code necessary to build the .NET product available in one repository and identified by a single commit.
• The VMR also aims to become the place from which we release and service future versions of .NET to reduce the complexity of the product construction process. This should allow our partners and and 3rd parties to easily build, test and modify .NET using their custom infrastructure as well as make the process available to the community.
• Lastly, we hope to solve other problems that the current multi-repo setup brings:
  • Enable the standard down-/up-stream open-source model.
  • Fulfill requirements of .NET distro builders such as RedHat or Canonical to natively include .NET in their distribution repositories.
  • Simplify scenarios such as client-run testing of bug fixes and improvements. The build should work in an offline environment too for certain platforms.
  • Enable developers to make and test changes spanning multiple repositories.
  • More efficient pipeline for security fixes during the CVE pre-disclosure process.

We will achieve these goals while keeping active coding work in the separate repos where it happens today. For example: ASP.NET features will continue to be developed in dotnet/aspnetcore and CLR features will be continue to be developed in dotnet/runtime. Each of these repos have their own distinct communities and processes, and aggregating development into a true mono-repo would work against that. Hence, the "virtual" monolithic repo: the VMR gives us the simplicity of a mono-repo for building and servicing the product, while active development of components of that product stays in its various existing repos. The day to day experience for typical contributors will not change.

Limitations

This is a work-in-progress. There are considerable limitations to what is possible at the moment. For an extensive list of current limitations, please see Temporary Mechanics.
See the Unified Build roadmap for more details.

Supported platforms

• 8.0
  • source-build configuration on Linux
• 9.0+ (WIP)
  • source-build configuration on Linux
  • non-source-build configuration on Linux, Mac, and Windows

For the latest information about Source-Build support for new .NET versions, please check our GitHub Discussions page for announcements.

Code flow

For the time being, the source code only flows one way - from the development repos into the VMR. More details on this process:

• Source Synchronization Process
• Synchronization Based on Declared Dependencies
• Moving Code and Dependencies between the VMR and Development Repos

We expect the code flow to start working both ways in the .NET 9 timeframe. See the Unified Build roadmap for more details.

Contribution

At this time, the VMR will not accept any changes and is a read-only mirror of the development repositories only. Please, make the changes in the respective development repositories (e.g., dotnet/runtime or dotnet/sdk) and they will get synchronized into the VMR automatically.

Dev instructions

Please note that this repository is a work-in-progress and there are some usability issues connected to this. These can be nuisances such as some checked-in files getting modified by the build itself and similar. For the latest information about Source-Build support, please watch for announcements posted on our GitHub Discussions page.

Prerequisites

The dependencies for building can be found here. In case you don't want to / cannot prepare your environment per the requirements, consider using Docker.

Building

1. Clone the repository

   git clone https://github.com/dotnet/dotnet dotnet-dotnet
   cd dotnet-dotnet

2. Build the .NET SDK

   Choose one of the following build modes:

   • Microsoft based build

     For Unix:

     ./build.sh --clean-while-building

     For Windows:

     .\build.cmd -cleanWhileBuilding

   • Building from source

     # Prep the source to build on your distro.
     # This downloads a .NET SDK and a number of .NET packages needed to build .NET from source.
     ./prep-source-build.sh
     
     # Build the .NET SDK
     ./build.sh -sb --clean-while-building

   The resulting SDK is placed at artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz (for Unix) or artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].zip (for Windows).

3. (Optional) Unpack and install the .NET SDK

   For Unix:

   mkdir -p $HOME/dotnet
   tar zxf artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz -C $HOME/dotnet
   ln -s $HOME/dotnet/dotnet /usr/bin/dotnet

   For Windows:

   mkdir %userprofile%\dotnet
   tar -xf artifacts/assets/Release/dotnet-sdk-9.0.100-[your RID].zip -C %userprofile%\dotnet
   set "PATH=%userprofile%\dotnet;%PATH%"

   To test your built SDK, run the following:

   dotnet --info

Note

Run ./build.sh --help (for Unix) or .\build.cmd -help (for Windows) to see more information about supported build options.

Building using Docker

You can also build the repository using a Docker image which has the required prerequisites inside. The example below creates a Docker volume named vmr and clones and builds the VMR there.

docker run --rm -it -v vmr:/vmr -w /vmr mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream9
git clone https://github.com/dotnet/dotnet .

# - Microsoft based build
./build.sh --clean-while-building

# - Building from source
./prep-source-build.sh && ./build.sh -sb --clean-while-building

mkdir -p $HOME/.dotnet
tar -zxf artifacts/assets/Release/dotnet-sdk-9.0.100-centos.9-x64.tar.gz -C $HOME/.dotnet
ln -s $HOME/.dotnet/dotnet /usr/bin/dotnet

Codespaces

You can also utilize GitHub Codespaces where you can find preset containers in this repository.

Building from released sources

You can also build from sources (and not from a context of a git repository), such as the ones you can acquire from a dotnet/dotnet release. In this case, you need to provide additional information which includes the original repository and commit hash the code was built from so that the SDK can provide a better debugging experience (think the Step into.. functionality). Usually, this means the dotnet/dotnet repository together with the commit the release tag is connected to.

In practice, this means that when calling the main build script, you need to provide additional arguments when building outside of a context of a git repository.
Alternatively, you can also provide a manifest file where this information can be read from. This file (release.json) can be found attached with the dotnet/dotnet release.

Synchronizing code into the VMR

Sometimes you want to make a change in a repository and test that change in the VMR. You could of course make the change in the VMR directly (locally, as the VMR is read-only for now) but in case it's already available in your repository, you can synchronize it into the VMR (again locally).

To do this, you can either start a dotnet/dotnet Codespace - you will see instructions right after it starts. Alternatively, you can clone the repository locally and use the vmr-sync.sh or vmr-sync.ps1 script to pull your changes in. Please refer to the documentation in the script for more details.

List of components

The full list of components synchronized into the VMR is here (Components.md).

The repository also contains a JSON manifest listing all components in a machine-readable format.

Filing Issues

This repo does not accept issues as of now. Please file issues to the appropriate development repos. For issues with the VMR itself, please use the source-build repository.

Useful Links

• Design documentation for the VMR - a set of documents describing the high-level design and the why's and how's
  • Design and Operation
  • Upstream/Downstream Relationships
  • Code and Build Workflow
  • Strategy for Managing External Source Dependencies
• .NET Source-Build
• What is .NET

.NET Foundation

.NET Runtime is a .NET Foundation project.

License

.NET is licensed under the MIT license.

xdt's Issues

Target NetStandard1.3 instead of netstandard2.0

Fix the dependencies and target netstandard1.3 instead of netstandard1.5

Enable source-build pre-build detection

Part of dotnet/source-build#3017

Enable source-build pre-build detection on the current repository and resolve any pre-build issues discovered

Specify all source-built dependencies and ensure they are regularly updated

Overview

Source-build works today by building a repository, gathering the versions of the outputs, and creating a property file with those versions. This property file is then fed to downstream repositories, which import it after their eng/Versions.props. The resulting behavior is that the downstream repo overrides all dependency versions specified in eng/Versions.props with the versions built from source. This behavior is not the same as the Maestro dependency-flow based approach. Maestro only updates properties for dependencies that are specified in eng/Version.Details.xml.

This causes the following difference: Say that dotnet/aspnetcore has a Microsoft.Net.Compilers.Toolset dependency. It codes a property for that dependency's version in eng/Versions.props as MicrosoftNetCompilersToolsetVersion, with an older version of the package.

• When building via source-build, roslyn builds first, and aspnetcore will pick up the new version of Microsoft.Net.Compilers.Toolset, even though it doesn't want to.
• When building officially, the non-latest version will be used.

This difference tends to cause build breaks in source-build. The version bump may be significant and require repo reaction. This is not ideal and not sustainable.

One option would be to only override those versions that are specified in eng/Version.Details.xml. This would more closely align the source-build and current official builds. The huge downside is that this will cause an explosion of ref packs. And in some cases, we would be building against old versions but actually executing against newer ones. Non-ideal.

In summary, the goals are:

• Reduce source-build build breaks
• Avoid a large increase in ref packs.
• Ensure that it is possible to freeze on old versions of a specific component if absolutely necessary

Work Required

This work is dependent on dotnet/source-build#2982.

In this repo, do the following

• Obtain a report of un-specified dependencies in this repo. This issue will be updated with instructions on how to do so after dotnet/source-build#2982 is complete.
• Specify all dependencies classified as missing in the report in the eng/Version.Details.xml file.
• Upgrade dependencies regularly via Maestro dependency flow mechanisms, or pin dependencies that should stay the same, with justification.

On 3/4 we're going to rename the default branch in this repository to main. For more details, see [our earlier announcement](https://github.com/dotnet/announcements/issues/172).

new versions missing from nuget.org

last published in 2020 https://www.nuget.org/packages/Microsoft.Web.Xdt/

any plans on pushing 5 and 6 to nuget.org?

Update the Project site and source repository to point to a public repo instead of xdt private repo

The project site information and source information points to this repo which is private. Change this to a public repo (may be websdk or one of the tooling public repos) so that users can file issues/ask questions.

Add more unit tests & create samples for how to use the Package.

TransformXml no longer preserves line breaks in VS2019

Created on behalf of this developer community ticket - https://developercommunity.visualstudio.com/content/problem/711827/transformxml-no-longer-preserves-line-breaks-in-vs-1.html

Microsoft.Web.Xdt 3.0.0 has a lower fileversion than 2.X

Some context:

We use the Microsoft.Web.Xdt package in our product to transform XMLs and after a update from version 2.1.2 to 3.0.0 we found some issues regarding the file version.

In the 2.X days we used a bindingRedirect like this:

      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Web.XmlTransform" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-15.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>

After the update (and with the bindingRedirect in place) the xml transform resulted in an error. I resolved this issue by removing the unnessary bindingRedirect.

So far so good, but we distribute our application via MSI and I guess this could be problematic in update scenarios.

AFAIK the default MSI update behavior is to check if the file version has changed and if a newer file version is found, the .dll will be replaced. But with the 3.0 NuGet package this is not the case anymore:

(Left 2.X package, right 3.X package)

Currently, to make it "safe" I need to tell my clients to uninstall and reinstall the product, because an update probably won't update the Microsoft.Web.XmlTransform.dll.

I'm not sure if your file version "3.X" has a deeper meaning, but for compatibility it would be cool to specify a higher version than 15.

Upgrade to .NET 9 SDK

This repo should be targeting the .NET 9 SDK from the global.json file. This is a requirement necessary for .NET source build in order to target the net9.0 TFM. If necessary, this change can be made after the first public preview release.

Related to dotnet/source-build#3770

Use new Arcade SDK publishing infrastructure

This repository uses a very old form of the Arcade SDK publishing infrastructure. That infra is planned to be removed from the Arcade SDK by the end of the first half of January 2020. The repository build definition(s) need to be updated to use the new publishing infrastructure, otherwise its official builds will fail to publish. It’s a simple process, documentation about it is here. If you have any question or problem please reach out to @JohnTortugo, @riarenas, @mmitche or .NET Core Engineering

Once we remove the old infrastructure you might not be able to get an updated Arcade SDK (or Arcade updates at all) if you’re not using the new publishing infrastructure.

TFM trimming, and elimination of net4* targeting in Linux source-build

Description

Trim away unneeded TFMs, and conditionalize away non-applicable TFMs for Linux source build.

Requires dotnet/source-build#3013. source-build net4* targeting elimination required dotnet/source-build#2974

The purpose of this operation is twofold:

• Reduce the number of TFMs present overall in .NET's build, potentially improving build time.
• Eliminate framework TFMs, drastically reducing the size of the source build repository size.

When TFMs other than the latest .NET Core TFM are targeted in source-build, the reference assets for that TFM must be provided via a source build reference package. This is a 'text only' package that is assembled from IL at the beginning of the build to satisfy the required reference assets. In Microsoft's build, these would be restored from the internet, but in source build this is not allowed.

These SBRPs are large. All told, the total size contribution for just the netframework 4* TFMs is around 3GB. Not only does this have a significant contribution to the source build repo size, but it also affects build time. For Linux source build, we spend large amounts of time building assets that are simply thrown away at the end.

An 'ideal' would be a single TFM used throughout all of the .NET build, source build or otherwise. This is not possible. There are legitimate reasons to target older TFMs, especially netstandard. However, we should be able to justify each additional TFM.

To complete this work, repositories will use the tooling provided via dotnet/source-build#2901 to identify the TFMs in use in their repository. For each TFM, they should do the following:

• If the TFM can be eliminated, or upgraded to latest (e.g. target the latest netcore explicitly, or use net472), do so. netframework TFM removal for tools and internal code is likely low hanging fruit.
• Requires dotnet/source-build#2974 If the TFM is a framework TFM, it should be conditionalized out in source build for Linux.

It is a goal that the form of this conditionalization remain consistent across projects, and used relatively sparingly. How this conditionalization should be done will be designed in dotnet/source-build#2974. Once that is done, this issue will be updated with implementation details.

Add build scripts to create, sign & generate NuGet package for XDT.

Unified Build control update

Description

This issue is dependent on dotnet/source-build#2974 and is a placeholder for .NET 8 required work in product repositories.

Instructions for implementation will be updated once dotnet/source-build#2974 but will likely be a mechanical shift of properties in this repo. In a general sense, this is scaffolding work for expanding the set of source-built platforms beyond Linux.

CA2022 error in src/Microsoft.Web.XmlTransform/XmlFileInfoDocument.cs

New SDK introduces new warnings that turn into errors in VMR build - see dotnet/installer#19145 (comment)

##[error]/vmr/src/xdt/src/Microsoft.Web.XmlTransform/XmlFileInfoDocument.cs(93,17): error CA2022: (NETCORE_ENGINEERING_TELEMETRY=Build) Avoid inexact read with 'System.IO.Stream.Read(byte[], int, int)' (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca2022)

I'm going to patch to disable this warning, in VMR builds. However this needs to be properly fixed.

Is it possible to use this in the build process?

I'm sorry if this is more of a StackOverflow styled question than a project issue, but I'm trying to find any documentation on this project and can't find much.

Currently in the process of porting my .NET Framework projects to .NET5. We used XDT transformations on both web.config and app.config files and wish to keep doing it (until we move everything to the new json configuration format).

Previously, we would reference Microsoft.Web.Publishing.Tasks.dll in a build task in the csproj file this way:

  <UsingTask TaskName="TransformXml" AssemblyFile="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\Web\Microsoft.Web.Publishing.Tasks.dll" />
  <Target Name="App_config_AfterCompile" AfterTargets="AfterCompile" Condition="Exists('App.$(Configuration).config')">
    <!--Generate transformed app config in the intermediate directory-->
    <TransformXml Source="App.config" Destination="$(IntermediateOutputPath)$(TargetFileName).config" Transform="App.$(Configuration).config" />
    <!--Force build process to use the transformed configuration file from now on.-->
    <ItemGroup>
      <AppConfigWithTargetPath Remove="App.config" />
      <AppConfigWithTargetPath Include="$(IntermediateOutputPath)$(TargetFileName).config">
        <TargetPath>$(TargetFileName).config</TargetPath>
      </AppConfigWithTargetPath>
    </ItemGroup>
  </Target>

That publishing task DLL is now only available in Visual Studio, and not in the .NET SDK and can't be used in the "dotnet.exe" tools pipeline.

I found an alternative that works: dotnet-transform-xdt nuget github. It provides a DotNetCliToolReference nuget that you can include in your project references, and create a target in your csproj file that call "dotnet transform-xdt" and it works fine. Unfortunately, the author of this project deprecated it, told me it was only temporary in the early stages of .NET Core and now that dotnet/xdt exists officially to use that instead.

Now I'm trying to use dotnet/xdt in my build process but can't seems to do it. The only usage i can find is how to create an application that would launch a transformation, but nothing that plug itself in the build process. I tried to keep the previous method of UsingTask/Target while referencing $(NuGetPackageRoot) up to Microsoft.Web.XmlTransform.dll for the AssemblyFile to no avail, the new DLL don't seems to expose a task called TransformXml.

So my question is, is this possible, and if not is it planned to be supported in the future?

Add loc support and generate resource dlls as part of build step

New Transform action "Upsert" highly appretiated

Hi there,

i love this transform since it was developed in the SlowCheetah project.
but 1 thing is still missing in my opinion:

xdt:Transform="Upsert"

which means update or insert in words or maybe fully written UpdateOrInsert to follow the line of InsertIfMissing

what do you think? will it be possible?

Is there a sample project or other usage documentation available

Not sure if this is the correct location to ask usage related questions but I did not find any other location to do that. I am trying to run a web.config transformation using commands in the .csproj file of my project.

Project Details
AspNetCore 2.1.x project targeting .Net Framework 4.6.1.

Use Case
I want to change the value of an attribute in the generated web.config file for my project. I use dotnet publish to publish the project for IIS deployment and I want directives in the csproj to trigger transformation of the generated web.config file. I am trying to add a nuget reference to project and leverage it for the transformation in a way similar to (1) https://github.com/nil4/dotnet-transform-xdt#-project-level-dotnet-transform-xdt-tool- or (2) https://stackoverflow.com/a/10506476/4764075

Can this library be leveraged to this end. If so, are there any examples available for that. If not, since this has been quoted as the library responsible for transforming web.config by Visual Studio and .NET CLI, would you recommend some other ways to approach solving this problem?

Fix the linux part of the Xdt build

https://github.com/aspnet/xdt/blob/78dfa1f627c501a3e601db9b6a07022e8da9abf8/azure-pipelines.yml#L117-L138