dotnet / azcopynet

.NET Library to seamlessly leverage AzCopy

dotnet/dotnet - Home of the .NET VMR

This repository is a Virtual Monolithic Repository (VMR) which includes all the source code and infrastructure needed to build the .NET SDK.

What this means:

• Monolithic - a join of multiple repositories that make up the whole product, such as dotnet/runtime or dotnet/sdk.
• Virtual - a mirror (not replacement) of product repos where sources from those repositories are synchronized into.

In the VMR, you can find:

• source files of each product repository which are mirrored inside of their respective directories under src/,
• tooling that enables building the whole .NET product from source on Linux platforms,
• small customizations, in the form of patches, applied on top of the original code to make the build possible,
• [in future] E2E tests for the whole .NET product.

Just like the development repositories, the VMR will have a release branch for every feature band (e.g. release/8.0.1xx). Similarly, VMR's main branch will follow main branches of product repositories (see Synchronization Based on Declared Dependencies).

More in-depth documentation about the VMR can be found in VMR Design And Operation. See also dotnet/source-build for more information about our whole-product source-build.

Goals

• The main purpose of the dotnet/dotnet repository is to have all source code necessary to build the .NET product available in one repository and identified by a single commit.
• The VMR also aims to become the place from which we release and service future versions of .NET to reduce the complexity of the product construction process. This should allow our partners and and 3rd parties to easily build, test and modify .NET using their custom infrastructure as well as make the process available to the community.
• Lastly, we hope to solve other problems that the current multi-repo setup brings:
  • Enable the standard down-/up-stream open-source model.
  • Fulfill requirements of .NET distro builders such as RedHat or Canonical to natively include .NET in their distribution repositories.
  • Simplify scenarios such as client-run testing of bug fixes and improvements. The build should work in an offline environment too for certain platforms.
  • Enable developers to make and test changes spanning multiple repositories.
  • More efficient pipeline for security fixes during the CVE pre-disclosure process.

We will achieve these goals while keeping active coding work in the separate repos where it happens today. For example: ASP.NET features will continue to be developed in dotnet/aspnetcore and CLR features will be continue to be developed in dotnet/runtime. Each of these repos have their own distinct communities and processes, and aggregating development into a true mono-repo would work against that. Hence, the "virtual" monolithic repo: the VMR gives us the simplicity of a mono-repo for building and servicing the product, while active development of components of that product stays in its various existing repos. The day to day experience for typical contributors will not change.

Limitations

This is a work-in-progress. There are considerable limitations to what is possible at the moment. For an extensive list of current limitations, please see Temporary Mechanics.
See the Unified Build roadmap for more details.

Supported platforms

• 8.0
  • source-build configuration on Linux
• 9.0+ (WIP)
  • source-build configuration on Linux
  • non-source-build configuration on Linux, Mac, and Windows

For the latest information about Source-Build support for new .NET versions, please check our GitHub Discussions page for announcements.

Code flow

For the time being, the source code only flows one way - from the development repos into the VMR. More details on this process:

• Source Synchronization Process
• Synchronization Based on Declared Dependencies
• Moving Code and Dependencies between the VMR and Development Repos

We expect the code flow to start working both ways in the .NET 9 timeframe. See the Unified Build roadmap for more details.

Contribution

At this time, the VMR will not accept any changes and is a read-only mirror of the development repositories only. Please, make the changes in the respective development repositories (e.g., dotnet/runtime or dotnet/sdk) and they will get synchronized into the VMR automatically.

Dev instructions

Please note that this repository is a work-in-progress and there are some usability issues connected to this. These can be nuisances such as some checked-in files getting modified by the build itself and similar. For the latest information about Source-Build support, please watch for announcements posted on our GitHub Discussions page.

Prerequisites

The dependencies for building can be found here. In case you don't want to / cannot prepare your environment per the requirements, consider using Docker.

Building

1. Clone the repository

   git clone https://github.com/dotnet/dotnet dotnet-dotnet
   cd dotnet-dotnet

2. Build the .NET SDK

   Choose one of the following build modes:

   • Microsoft based build

     For Unix:

     ./build.sh --clean-while-building

     For Windows:

     .\build.cmd -cleanWhileBuilding

   • Building from source

     # Prep the source to build on your distro.
     # This downloads a .NET SDK and a number of .NET packages needed to build .NET from source.
     ./prep-source-build.sh
     
     # Build the .NET SDK
     ./build.sh -sb --clean-while-building

   The resulting SDK is placed at artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz (for Unix) or artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].zip (for Windows).

3. (Optional) Unpack and install the .NET SDK

   For Unix:

   mkdir -p $HOME/dotnet
   tar zxf artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz -C $HOME/dotnet
   ln -s $HOME/dotnet/dotnet /usr/bin/dotnet

   For Windows:

   mkdir %userprofile%\dotnet
   tar -xf artifacts/assets/Release/dotnet-sdk-9.0.100-[your RID].zip -C %userprofile%\dotnet
   set "PATH=%userprofile%\dotnet;%PATH%"

   To test your built SDK, run the following:

   dotnet --info

Note

Run ./build.sh --help (for Unix) or .\build.cmd -help (for Windows) to see more information about supported build options.

Building using Docker

You can also build the repository using a Docker image which has the required prerequisites inside. The example below creates a Docker volume named vmr and clones and builds the VMR there.

docker run --rm -it -v vmr:/vmr -w /vmr mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8
git clone https://github.com/dotnet/dotnet .

# - Microsoft based build
./build.sh --clean-while-building

# - Building from source
./prep-source-build.sh && ./build.sh -sb --clean-while-building

mkdir -p $HOME/.dotnet
tar -zxf artifacts/assets/Release/dotnet-sdk-9.0.100-centos.8-x64.tar.gz -C $HOME/.dotnet
ln -s $HOME/.dotnet/dotnet /usr/bin/dotnet

Codespaces

You can also utilize GitHub Codespaces where you can find preset containers in this repository.

Building from released sources

You can also build from sources (and not from a context of a git repository), such as the ones you can acquire from a dotnet/dotnet release. In this case, you need to provide additional information which includes the original repository and commit hash the code was built from so that the SDK can provide a better debugging experience (think the Step into.. functionality). Usually, this means the dotnet/dotnet repository together with the commit the release tag is connected to.

In practice, this means that when calling the main build script, you need to provide additional arguments when building outside of a context of a git repository.
Alternatively, you can also provide a manifest file where this information can be read from. This file (release.json) can be found attached with the dotnet/dotnet release.

Synchronizing code into the VMR

Sometimes you want to make a change in a repository and test that change in the VMR. You could of course make the change in the VMR directly (locally, as the VMR is read-only for now) but in case it's already available in your repository, you can synchronize it into the VMR (again locally).

To do this, you can either start a dotnet/dotnet Codespace - you will see instructions right after it starts. Alternatively, you can clone the repository locally and use the vmr-sync.sh or vmr-sync.ps1 script to pull your changes in. Please refer to the documentation in the script for more details.

List of components

The full list of components synchronized into the VMR is here (Components.md).

The repository also contains a JSON manifest listing all components in a machine-readable format.

Filing Issues

This repo does not accept issues as of now. Please file issues to the appropriate development repos. For issues with the VMR itself, please use the source-build repository.

Useful Links

• Design documentation for the VMR - a set of documents describing the high-level design and the why's and how's
  • Design and Operation
  • Upstream/Downstream Relationships
  • Code and Build Workflow
  • Strategy for Managing External Source Dependencies
• .NET Source-Build
• What is .NET

.NET Foundation

.NET Runtime is a .NET Foundation project.

License

.NET is licensed under the MIT license.

azcopynet's Issues

missing support for arm

Hey,

noticed there is no support for arm.
did i missed it ?
if i did not, are you planning to add support for arm ?

Thanks :)

AzCopy.WinX64 1.0.2 contains wrong azcopy executable which causes System.Exception

AzCopy.WinX64 1.0.2 contains azcopy not azcopy.exe which causes exception on Windows 10 running locally in a Console Application in VS 2019.

Running project with AzCopy.Client 1.0.2 and AzCopy.WinX64 1.0.2 packages causes an exception

Can't find azcopy. Make sure you install azcopy and set its path to $AZCOPYPATH on your system, or use one of the following nuget package: AzCopy.WinX64, AzCopy.LinuxX64, AzCopy.OsxX64.

Worked around by renaming the file azcopy.exe in the build output folder. Can then copy the into a project, set the file's Copy to output Directory property to Copy Always which then allows the build to run but defeats the purpose of the extra package in the first place.

Package is not available on nuget?

I tried searching for this package on nuget.org but can't find it. The nuget package links in the ReadMe reference a private nuget feed at https://dev.azure.com/xiaoyuz0315/BigMiao.

Is there a reason why this isn't on nuget.org or am I just missing something?
We can't use a private nuget, which is unfortunate as the package looks great.

Setting of key and secret to copy from AWS to Azure is unclear from the examples

To copy from AWS to Azure you need to set the Access Key Id and Secret. When using cmd you can apply this before executing azcopy. How would this be done in the current solution? I could not find any tests or info om this. I am not sure how to set it. Because it seems like the command is executed before setting the variables. Or am i misunderstanding something?

This is how i used to do it executing cmd.exe from code.
process.StandardInput.WriteLine($"set AWS_ACCESS_KEY_ID={S3_ACCESS_KEY_ID}");
process.StandardInput.WriteLine($"set AWS_SECRET_ACCESS_KEY={S3_SECRET_ACCESS_KEY}");

Process_OutputDataReceived trigger an exception for JobsListAsync

With azcopy.exe version 10.12.1, JobsListAsync trigger an exception in Process_OutputDataReceived:

Newtonsoft.Json.JsonSerializationException
HResult=0x80131500
Message=Required property 'JobStatus' not found in JSON. Path '', line 1, position 13776.
Source=Newtonsoft.Json
StackTrace:
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EndProcessProperty(Object newObject, JsonReader reader, JsonObjectContract contract, Int32 initialDepth, JsonProperty property, PropertyPresence presence, Boolean setDefaultValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value)
at AzCopy.Client.AZCopyClient.Process_OutputDataReceived(Object sender, DataReceivedEventArgs e) in D:\Temp\AzCopyNet-main\src\AzCopy.Client\AZCopyClient.cs:line 228

This exception was originally thrown at this call stack:
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EndProcessProperty(object, Newtonsoft.Json.JsonReader, Newtonsoft.Json.Serialization.JsonObjectContract, int, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PropertyPresence, bool)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(object, Newtonsoft.Json.JsonReader, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, string)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(Newtonsoft.Json.JsonReader, System.Type, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, object)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(Newtonsoft.Json.JsonReader, System.Type, bool)
Newtonsoft.Json.JsonSerializer.DeserializeInternal(Newtonsoft.Json.JsonReader, System.Type)
Newtonsoft.Json.JsonSerializer.Deserialize(Newtonsoft.Json.JsonReader, System.Type)
Newtonsoft.Json.JsonConvert.DeserializeObject(string, System.Type, Newtonsoft.Json.JsonSerializerSettings)
Newtonsoft.Json.JsonConvert.DeserializeObject(string, Newtonsoft.Json.JsonSerializerSettings)
Newtonsoft.Json.JsonConvert.DeserializeObject(string)
AzCopy.Client.AZCopyClient.Process_OutputDataReceived(object, System.Diagnostics.DataReceivedEventArgs) in AZCopyClient.cs

The content of message.MessageContent:

{
  "ErrorMessage": "",
  "JobIDDetails": [
    {
      "JobId": "7fde53c1-e158-0c44-4331-dd337533a865",
      "CommandString": "copy https://x.blob.core.windows.net/files/x.zip?se=2021-10-15t19%3A47%3A11z\u0026sig=REDACTED\u0026sp=rl\u0026sr=c\u0026sv=2020-04-08 C:\\Users\\p\\Downloads\\x.zip --output-type=json --cancel-from-stdin --overwrite=prompt --check-md5 FailIfDifferent --from-to=BlobLocal --recursive --trusted-microsoft-suffixes= --log-level=INFO ",
      "StartTime": 1631735231393907100,
      "JobStatus": "Completed"
    },
    {
      "JobId": "1d5e2bd7-8a5d-6d4a-77c3-5e2562ced9dd",
      "CommandString": "remove https://x.blob.core.windows.net/files/x.zip?se=2021-10-15t19%3A46%3A07z\u0026sig=REDACTED\u0026sp=rdl\u0026sr=c\u0026sv=2020-04-08 --output-type=json --cancel-from-stdin --recursive --trusted-microsoft-suffixes= --log-level=INFO ",
      "StartTime": 1631735170230821700,
      "JobStatus": "Completed"
    },
    {
      "JobId": "1d7b2b65-b5c5-b64b-592a-0d50fceec9fe",
      "CommandString": "copy D:\\Sources\\x.zip https://x.blob.core.windows.net/files/x.zip?se=2021-10-15t19%3A45%3A49z\u0026sig=REDACTED\u0026sp=rwl\u0026sr=c\u0026sv=2020-04-08 --output-type=json --cancel-from-stdin --overwrite=prompt --from-to=LocalBlob --blob-type Detect --follow-symlinks --put-md5 --follow-symlinks --recursive --trusted-microsoft-suffixes= --log-level=INFO ",
      "StartTime": 1631735149828609400,
      "JobStatus": "Completed"
    }
  ]
}

So, the output is an array, but the code tries to deserialize a single object:

Package links in README.md are dead

Clicking any package or nightly build links in README leads to HTTP 503 and following message:
TF24668: The following team project collection is stopped: xiaoyuz0315. Start the collection and then try again. Administrator Reason: This Azure DevOps organization has been deactivated due to inactivity. For more information please visit aka.ms/adofootprintreduction

AzCopy download throws exception.