sitecoreunicorn / rainbow

An advanced serialization and comparison system for Sitecore. Designed for easy merging with YAML-based serialization, and a generic merge/deserialization framework. Works well with Unicorn, or by itself.

Rainbow

An advanced serialization library for Sitecore 7 and above. Rainbow is designed to be a complete replacement for the Sitecore serialization format and filesystem organization, as well as enabling cross-source item comparison. It is a pure code library that comes with no UI of any kind, that is designed to be used with other libraries that use its serialization services with their own UIs. Libraries that consume Rainbow - such as Unicorn - gain the ability to abstract themselves from serialization details. Libraries that extend Rainbow can add new serialization formats, new places to store serialized items, and new ways to organize them.

Rainbow Features

Universal Item Data and Data Stores

Rainbow implements a set of interfaces that wrap the structure of a Sitecore item - item, version, and field. These interfaces provide a universal language that all Rainbow data stores can implement against. You could get an IItemData from Sitecore and write it out to disk as a YAML formatted item. You could get an IItemData from a web service, and deserialize it into a Sitecore database. You could construct an IItemData programmatically, and serialize it to a Sitecore database. Implementations of IDataStore provide places to store item data. It's completely universal, and everything Rainbow does revolves around these abstractions.

YAML-based serialization formatter improves the storage format for serialized items

• This post goes into more detail about the hows and whys of the YAML serializer
• The format is valid YAML. YAML is a language that is designed to store object graphs in a human readable fashion. It uses significant whitespace and indentation to denote data boundaries. Note: only a subset of the YAML spec is allowed for performance reasons.
• Any type of endline support. Yes, even \r because one of the default Sitecore database items uses that in its text! No more .gitattributes needed.
• No more Content-Length on fields that requires manual recalculation after merge conflicts
• Multilists are stored multi-line so fewer conflicts can occur
• XML fields (layout, rules) are stored pretty-printed so that fewer conflicts can occur
• Customize how fields are stored when serialized yourself with Field Formatters

Serialization File System (SFS) storage hierarchy

• This post goes into more detail about SFS
• Human readable file hierarchy
• Extremely long item name support
• Unlimited path length support
• Supports non-unique paths (two items under the same parent with the same name), while keeping it human-readable
• Stores each included subtree in its own hierarchy, reducing file name length
• You can plug in the serialization formatter you desire - such as the YAML provider - to format items how you want in the tree

Deserialize abstract items into Sitecore with the Sitecore storage provider

• Turn Sitecore items into IItemData with the ItemData class
• Deserialize an IItemData instance into a Sitecore item with DefaultDeserializer (used via SitecoreDataStore)
• Query the Sitecore tree in abstract with SitecoreDataStore, as if it were any other serialization store

Item comparison APIs

• Compare any two IItemData instances regardless of source
• Customize comparison for field types or specific fields with Field Comparers
• Get a complete readout of changes as an object model

Improvements

Improvements are in comparison to Sitecore serialization and the functionality in Unicorn 2.

• Deleting template fields will no longer cause errors on deserialization for items that are serialized with a value for the deleted field (e.g. standard values)
• Deserialization knows how to properly change field sharing or field versioning and port existing data to the new sharing setting

Rainbow Organization

Rainbow consists of several projects:

• The core Rainbow project contains core interfaces and components that most all serialization components might need, for example SFS, item comparison tools, field formatters, and item filtering
• Rainbow.Storage.Yaml implements serializing and deserializing items using a YAML-based format. This format is ridiculously easier to read and merge than standard Sitecore serialization format, lacking any content length attributes, supporting any type of newline characters, and supporting pretty-printing field values (e.g. multilists, layout) for simpler merging when conflicts occur.
• Rainbow.Storage.Sc implements a data store using the Sitecore database. This can be used to read and write items to Sitecore using the IDataStore interface.

Using Rainbow with TFS

Rainbow by default allows all characters that Windows allows to be in item filenames (when using SFS). Team Foundation Server does not allow files to be added which contain the $ character. To enable proper interaction with TFS, enable the Rainbow.TFS.config.example patch file that ships with the Rainbow NuGet package.

Extending Rainbow

Rainbow is designed to be loosely coupled. It's recommended that you employ a Dependency Injection framework (e.g. SimpleInjector) to make your life constructing Rainbow objects easier. Of course you can also construct objects without DI.

Rainbow has extensive unit and integration tests and hopefully easy to understand code, which serve as its living documentation. As of now, Rainbow has 90% test coverage and 220 tests. Unicorn, which uses Rainbow as a service, can also be a useful source of examples.

If you can't find what you're looking for you can find me on Twitter (@kamsar) or on Sitecore Community Slack.

rainbow's Issues

Windows reserved file names

I had an item named CON and I could not understand why Unicorn failed to serialize the item. Unicorn failed with:

Error while writing SFS item C:\mypath\Data\Unicorn\inRiver\Test\Home\CON.yml (Rainbow.Storage.SfsWriteException)
at Rainbow.Storage.SerializationFileSystemTree.WriteItem(IItemData item, String path) 
at Unicorn.SerializationHelper.DumpItemInternal(IItemData item, IPredicate predicate, ITargetDataStore targetDataStore) 
at Unicorn.SerializationHelper.<>c__DisplayClass5.b__4()
  FileStream will not open Win32 devices such as disk partitions and tape drives. Avoid use of "\\.\" in the path. (System.ArgumentException)
  at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) 
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy) 
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) 
  at Rainbow.Storage.SerializationFileSystemTree.WriteItem(IItemData item, String path)

Turns out CON is a reserved filename on Windows:

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9

Source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

I guess this should to be addressed in SFS?

Rainbow doesn't warn me when critical path length settings are different than expected

Do you want to request a feature or report a bug?
A quality of life improvement

What is the current behavior?
Today, it is possible for users (developers, usually) to change the Rainbow.config settings for SerializationFolderPathMaxLength and MaxItemNameLengthBeforeTruncation (presumably to accommodate for a long local root path) and forgetting that this setting needs to be the same across all environments and as such should never be touched locally.

If the current behavior is a bug, please provide the steps to reproduce.
Just change either of these settings without reserializing the repository. All sorts of item missing errors will occur, or cases of items not being serialized as expected. These issues are almost invisible and very hard to debug.

What is the expected behavior?
That Rainbow tries it's durn best to detect this and prevent this situation from going unchecked.

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.
Any

Rainbow and Sitecore 9

Do you want to request a feature or report a bug?
bug

What is the current behavior?
I'm working in a solution based on Sitecore 8.1 and I am trying to upgrade to Sitecore 9. In the Analysis process, I receive several warnings of Rainbow, for example:

<shortdescription>Type Rainbow.Settings.RainbowSitecoreSettings from assembly Rainbow uses the type that has got breaking changes.</shortdescription>

<description>The type Sitecore.Configuration.Settings defined in assmebly Sitecore.Kernel has got breaking changes. Custom type Rainbow.Settings.RainbowSitecoreSettings defined in assembly Rainbow uses the changed type in System.Int32 get_SfsSerializationFolderPathMaxLength() and might be affected.</description>

Is Rainbow compatible with Sitecore 9?

If yes, any reason why we are receiving these warnings? How we could avoid them?

In case we cannot overcome this we will remove Unicorn from our solution, install Sitecore 9 and then install Unicorn again but we would really prefer to solve this issue first.

What is the expected behavior?

There are no warnings in the Analysis result in the Update Installation Wizard

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.

Sitecore 8.1
Unicorn 4.0.3
Rainbow 2.0.0

Sync Errors with Rainbow 2.1.1 and SC Hotfix 400435-3 (Sc 9.3)

Hi Mark / Kam

I'm not quite sure if this is the correct channel to address this issue (because it could also be a Sitecore bug) but maybe you know something in this regard.

Thanks :)

Do you want to request a feature or report a bug?
A possible bug in Rainbow, Unicorn or Sitecore.

What is the current behavior?
If a (language) version is missing in the YML file (compared to the item in the database), Unicorn tries to delete this version (like it is supposed to) -> but this results in an Error like the following one:

25272 16:59:30 ERROR Failed to remove the item version. Item ID: {6974D84F-1212-4671-BAFE-0FDE31C3EE38}, version: 1, language: de-CH
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Source: Sitecore.Kernel
   at Sitecore.Data.DataSource.RemoveVersion(ID itemID, VersionUri uri)
   at Sitecore.Nexus.Data.DataCommands.RemoveVersionCommand.‪‪‬​‍‫‎‍‫‏‫‫‫‫‮​‫‎‬‭‍‌‮‮(Item )
   at Sitecore.Data.Engines.EngineCommand`2.Execute()
   at Sitecore.Data.Engines.DataEngine.RemoveVersion(Item item)

25272 16:59:30 INFO  [Unicorn]: * [D] de-CH#1
25272 16:59:30 ERROR Failed to remove the item version. Item ID: {6974D84F-1212-4671-BAFE-0FDE31C3EE38}, version: 1, language: de-CH
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Source: Sitecore.Kernel
   at Sitecore.Data.DataSource.RemoveVersion(ID itemID, VersionUri uri)
   at Sitecore.Nexus.Data.DataCommands.RemoveVersionCommand.‪‪‬​‍‫‎‍‫‏‫‫‫‫‮​‫‎‬‭‍‌‮‮(Item )
   at Sitecore.Data.Engines.EngineCommand`2.Execute()
   at Sitecore.Data.Engines.DataEngine.RemoveVersion(Item item)

25272 16:59:30 INFO  [Unicorn]: * [D] de-CH#1
25272 16:59:30 ERROR Failed to remove the item version. Item ID: {6974D84F-1212-4671-BAFE-0FDE31C3EE38}, version: 1, language: de-CH
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Source: Sitecore.Kernel
   at Sitecore.Data.DataSource.RemoveVersion(ID itemID, VersionUri uri)
   at Sitecore.Nexus.Data.DataCommands.RemoveVersionCommand.‪‪‬​‍‫‎‍‫‏‫‫‫‫‮​‫‎‬‭‍‌‮‮(Item )
   at Sitecore.Data.Engines.EngineCommand`2.Execute()
   at Sitecore.Data.Engines.DataEngine.RemoveVersion(Item item)

25272 16:59:30 INFO  [Unicorn]: * [D] de-CH#1
25272 16:59:30 ERROR ERROR: Some directories could not be loaded. (1 inner failures)
master:/sitecore/templates/Feature/Bookmarks/Configuration/Bookmark Configuration (6974d84f-1212-4671-bafe-0fde31c3ee38) (Unicorn.DeserializationAggregateException)
at Unicorn.Loader.DeserializeFailureRetryer.RetryAll(ISourceDataStore sourceDataStore, Action`1 retrySingleItemAction, Action`1 retryTreeAction)
   at Unicorn.Loader.SerializationLoader.LoadAll(IItemData[] rootItemsData, IDeserializeFailureRetryer retryer, IConsistencyChecker consistencyChecker, Action`1 rootLoadedCallback)
   at Unicorn.SerializationHelper.SyncTree(IConfiguration configuration, Action`1 rootLoadedCallback, Boolean runSyncStartPipeline, IItemData partialSyncRoot)
   at Unicorn.SerializationHelper.SyncConfigurations(IConfiguration[] configurations, IProgressStatus progress, ILogger additionalLogger)

INNER EXCEPTION
master:/sitecore/templates/Feature/Bookmarks/Configuration/Bookmark Configuration (6974d84f-1212-4671-bafe-0fde31c3ee38) (Rainbow.Storage.Sc.Deserialization.DeserializationException)
No stack trace available.
INNER EXCEPTION
Failed to paste item: /sitecore/templates/Feature/Bookmarks/Configuration/Bookmark Configuration (Rainbow.Storage.Sc.Deserialization.DeserializationException)
at Rainbow.Storage.Sc.Deserialization.DefaultDeserializer.Deserialize(IItemData serializedItemData, IFieldValueManipulator fieldValueManipulator)
   at Rainbow.Storage.Sc.SitecoreDataStore.Save(IItemData item, IFieldValueManipulator fieldValueManipulator)
   at Unicorn.Evaluators.SerializedAsMasterEvaluator.EvaluateUpdate(IItemData sourceItem, IItemData targetItem)
   at Unicorn.Loader.SerializationLoader.DoLoadItem(IItemData serializedItemData, IConsistencyChecker consistencyChecker)
   at Unicorn.Loader.DeserializeFailureRetryer.RetryAll(ISourceDataStore sourceDataStore, Action`1 retrySingleItemAction, Action`1 retryTreeAction)
INNER EXCEPTION
Object reference not set to an instance of an object. (System.NullReferenceException)
at Sitecore.Data.DataSource.RemoveVersion(ID itemID, VersionUri uri)
   at Sitecore.Nexus.Data.DataCommands.RemoveVersionCommand.‪‪‬​‍‫‎‍‫‏‫‫‫‫‮​‫‎‬‭‍‌‮‮(Item )
   at Sitecore.Data.Engines.EngineCommand`2.Execute()
   at Sitecore.Data.Engines.DataEngine.RemoveVersion(Item item)
   at Sitecore.Data.Managers.ItemProvider.RemoveVersion(Item item, SecurityCheck securityCheck)
   at Rainbow.Storage.Sc.Deserialization.DefaultDeserializer.PasteVersions(IItemData serializedItemData, Item targetItem, Boolean newItemWasCreated, List`1 softErrors, IFieldValueManipulator fieldValueManipulator)
   at Rainbow.Storage.Sc.Deserialization.DefaultDeserializer.Deserialize(IItemData serializedItemData, IFieldValueManipulator fieldValueManipulator)

If the current behavior is a bug, please provide the steps to reproduce.

• Make sure you have the SC Hotfix 400435-3 applied
• Delete a (language) version in a YML file and do a Unicorn sync.

What is the expected behavior?
Deletion of a (language) version should not cause an error.

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.

• Sitecore 9.3 with SC Hotfix 400435-3
• Unicorn 4.1.2-pre1
• Rainbow 2.1.1

Rainbow doesn't escape yaml specific symbols like : - &, etc.

If you symbol to the field that is used for structuring yaml and then serialize item with a Rainbow. It come up with invalid yaml file that can't be parsed by 3-party parsers.
As example, start field with a '-' followed by space or end field with ':', the you will have in file:
Value: - some value
I suggest to escape all values with doubleqoutes if they contain one of control symbols at least from Failsafe scheme:
-, :, [, ], {, }, !, ?

Error while writing SFS item

Maybe this i by design, I dont even know have the media item in question was created.

Trying to serialize via Unicorn the attach media item to disk gives the error below.

Sitecore version 8.1-update 1 Using Latest and greates Unicorn with Rainbow from github.

readmy_for_unicorn-1.0.0.0.zip

Error occurs in
Rainbow.Storage.Yaml.OutputModel.YamlFieldValue in the write yaml methodsometime the value is null but o guard against, maybe by design, but everthing else in the method is well protected.

`ERROR: One or more errors occurred. (System.AggregateException)
at Unicorn.SerializationHelper.DumpTreeInternal(IItemData root, IPredicate predicate, ITargetDataStore serializationStore, ISourceDataStore sourceDataStore, ILogger logger) in C:\Projects\Unicorn\src\Unicorn\SerializationHelper.cs:line 225
at Unicorn.SerializationHelper.DumpTree(IItemData item, IConfiguration[] configurations) in C:\Projects\Unicorn\src\Unicorn\SerializationHelper.cs:line 60
at Unicorn.ControlPanel.Pipelines.UnicornControlPanelRequest.ReserializeVerb.Process(IProgressStatus progress, ILogger additionalLogger) in C:\Projects\Unicorn\src\Unicorn\ControlPanel\Pipelines\UnicornControlPanelRequest\ReserializeVerb.cs:line 67
Error while writing SFS item c:\Websites\sc81rev151207\Data\Unicorn\all\ml\media library\readme.yml (Rainbow.Storage.SfsWriteException)
at Rainbow.Storage.SerializationFileSystemTree.WriteItem(IItemData item, String path) in C:\Projects\Rainbow\src\Rainbow\Storage\SerializationFileSystemTree.cs:line 372
at Rainbow.Storage.SerializationFileSystemTree.Save(IItemData item) in C:\Projects\Rainbow\src\Rainbow\Storage\SerializationFileSystemTree.cs:line 295
at Rainbow.Storage.SerializationFileSystemDataStore.Save(IItemData item) in C:\Projects\Rainbow\src\Rainbow\Storage\SerializationFileSystemDataStore.cs:line 50
at Unicorn.Data.ConfigurationDataStore.Save(IItemData item) in C:\Projects\Unicorn\src\Unicorn\Data\ConfigurationDataStore.cs:line 26
at Unicorn.SerializationHelper.DumpItemInternal(IItemData item, IPredicate predicate, ITargetDataStore targetDataStore) in C:\Projects\Unicorn\src\Unicorn\SerializationHelper.cs:line 232
at Unicorn.SerializationHelper.<>c__DisplayClass8_0.b__1() in C:\Projects\Unicorn\src\Unicorn\SerializationHelper.cs:line 184
Object reference not set to an instance of an object. (System.NullReferenceException)
at Rainbow.Storage.Yaml.YamlWriter.WriteMapInternal(String key, String value) in C:\Projects\Rainbow\src\Rainbow.Storage.Yaml\YamlWriter.cs:line 62
at Rainbow.Storage.Yaml.YamlWriter.WriteMap(String key, String value) in C:\Projects\Rainbow\src\Rainbow.Storage.Yaml\YamlWriter.cs:line 51
at Rainbow.Storage.Yaml.OutputModel.YamlFieldValue.WriteYaml(YamlWriter writer) in C:\Projects\Rainbow\src\Rainbow.Storage.Yaml\OutputModel\YamlFieldValue.cs:line 57
at Rainbow.Storage.Yaml.OutputModel.YamlItem.WriteYaml(YamlWriter writer) in C:\Projects\Rainbow\src\Rainbow.Storage.Yaml\OutputModel\YamlItem.cs:line 93
at Rainbow.Storage.Yaml.YamlSerializationFormatter.WriteSerializedItem(IItemData itemData, Stream outputStream) in C:\Projects\Rainbow\src\Rainbow.Storage.Yaml\YamlSerializationFormatter.cs:line 92
at Rainbow.Storage.SerializationFileSystemTree.WriteItem(IItemData item, String path) in C:\Projects\Rainbow\src\Rainbow\Storage\SerializationFileSystemTree.cs:line 366

`

FileSystemWatcher stops monitoring changes when root folder is deleted and recreated

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Have an empty repo; "master" with no root folder present (e.g. Unicorn). Then checkout develop and create the required root folder (e.g. Unicorn). Perform initial serialization. All predicates gets serialized fine.

Add to git and commit develop

checkout master

Log will fill with a lot of

5636 19:23:58 INFO Serialized item c:\Projects\TestDatastore\Unicorn\Core\sitecore\system\Dictionary\P\Publishing Items.yml deleted, reloading caches.
5636 19:23:58 WARN All caches have been cleared. This can decrease performance considerably.

(as expected, GIT is cleaning house)

Ultimately GIT will have cleaned out, including the root folder Unicorn and switch to master is complete.

Now repeat the operation. Checkout develop (all items come back in) and then master. No log messages to be found - the FSW no longer reports the file deletions. Presumably because it gave up watching when the root folder was destroyed.

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.

404-pre1

Updated items are not published

Hi

After upgrading to Rainbow.Storage.Sc 2.1.1.0 (Sitecore 9.2.0) we are facing an issue where items with updated field values are not published during Unicorn sync.

I have traced the issue to Rainbow/src/Rainbow.Storage.Sc/Deserialization/DefaultDeserializer.cs line 649 - if I am reading the code correctly, this should be a catch block, not a finally block?

Feature Request: Capability to use .zip files as .yml file store.

Idea

Currently, the Unicorn Rainbow Filesystem Provider uses the windows file system to store files.
It would be great, if the provider could also read and write .yml files from/to .zip files.

User Story

I'd like use .zip files to store Unicorn serialized .yml files on production systems.

• All Unicorn Configurations can choose whether to read their content from a .zip file or directly from the file system.
• Any Unicorn Configuration that uses a zip file storage is read-only (by default).
  Reserializing the content tree does not affect the .yml files in the .zip package.
• Any Unicorn Configurations which store items in a physical folder can still be serialized as .yml.
  This is helpful for content migration.

Concept

In a few words:

If you don't find a .yml file physically, look out for a .zip File containing that .yml file.

or

mount .yml files into tree as usual
mount .yml files contained in .zip files virtually into the tree.

f.e. if an item is serialized under C:\Projects\myproject\Unicorn\Feature\ModuleA\SomeLong\PathsWithin\ThatModule\MyItem.yml
then it's .yml contents can also be stored in Entry SomeLong\PathsWithin\ThatModule\MyItem.yml of a file named C:\Projects\myproject\Unicorn\Feature\ModuleA.zip.

Read only?

It would be ok if the .zip file storage is read-only, since we're mostly having issues with long paths on production systems where we just sync items into the database, not vice-versa. And if we re-sync from prod it's ok to use a pure .yml store.

Performance Impact

It also seems ok to me, that .zip file store could be a bit slower than direct file access, since it needs to read the .zip file and decompress the files. But that can be done with memory efficient "in-memory" Streams using the System.IO.Compression namespace offered by .NET Framework.
In the end, it might turn out faster to read a single file from a storage into memory and then process it in-memory, than querying the filesystem for thousands of small .yml files.

Related Issues

issue #27: long paths would be solved for most situations too.

Cannot serialize element

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Unicorn fails to serialize /sitecore/system/Settings/Rules/Definitions/Elements/Engagement Automation/Visibility/GeoIP visibility

If the current behavior is a bug, please provide the steps to reproduce.
Create a Unicorn config that includes /sitecore/system/Settings from the master database. Serialize. Sync.

What is the expected behavior?
That /sitecore/system/Settings/Rules/Definitions/Elements/Engagement Automation/Visibility/GeoIP visibility would be serialized without error

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.
Sitecore 8.2 (rev. 161221), Rainbow 1.4.1.0, Unicorn 3.3.2.0.

Sync Both Master and Web Database

What are the changes we need to make inorder to sync Both master and web database by default.
I will be calling the unicorn sync API command to sync and reserialize both the database on regular basics for backup.
Please provide that feature. in UI it shows checkbox to select the Database, but i want both the be selected by default.

BucketManager.MoveItemIntoBucket results in IOException: The directory is not empty. (sometimes)

Hi I am not sure if to put this in Unicorn or Rainbow.

When I call BucketManager.MoveItemIntoBucket to move a freshly created item I, sometimes, get an IOException: The directory is not empty. The reason I say sometimes is that sometimes it runs hundres of times and things are just fine but sometimes I get this exception:

Rainbow.Storage.SfsDeleteException: 'Error deleting SFS directory D:\xyzxyzxyz\Repo\sitecore\src\Sitecore.Data\Unicorn\xyz-xyz-xyz\master_xyz\65d63815-2f19-f285-9a10-2183297215ef'

IOException: The directory is not empty.

This exception was originally thrown at this call stack:
System.IO.__Error.WinIOError(int, string)
System.IO.Directory.DeleteHelper(string, string, bool, bool, ref Microsoft.Win32.Win32Native.WIN32_FIND_DATA)
System.IO.Directory.Delete(string, string, bool, bool)
Rainbow.Storage.SerializationFileSystemTree.RemoveWithoutChildren.AnonymousMethod__2()
Rainbow.ActionRetryer.Perform(System.Action)
Rainbow.Storage.SerializationFileSystemTree.RemoveWithoutChildren(Rainbow.Model.IItemMetadata)

Sitecore version: 9.0.1
Unicorn version: 4.0.4 (with transparent sync enabled)

Also something weird, normally items are located in:
D:\xyzxyzxyz\Repo\sitecore\src\Sitecore.Data\Unicorn\xyz-xyz-xyz\master_xyz\xyzxyzxyz xyz xyz\Content\Products

But sometimes it ends in the folder above.

Replace FakeItem with ProxyItem

The FakeItem class in the tests is duplicative with ProxyItem in the core.

Tests should migrate to use ProxyItem.

This also will allow Unicorn tests to not depend on Rainbow tests.

Feature request: Name comment additional to ID values

A serialized file contains a lot of ID's, sometimes it's not clear what that ID actually stands for, for example (this is a tree-list but it could be anything with ID's):
ID: "e391b526-d0c5-439d-803e-17512eae6222" Hint: Allowed Controls Type: TreelistEx Value: | {A21617EF-4786-46AE-A8EC-C0A5A10A74F9} {0F3395F7-5B5D-40BF-958F-AB738B5BCD55} {4ECF523A-03DF-4233-A73C-253C611B576B} {CF6F8563-D2DB-470F-94A8-46106BB449F8} {480D3D15-D70E-42AC-B0EF-4487FDB05D89}...

Now, when I merge a file like this using git, and multiple values have changed, I need to open-up Sitecore, enter the ID and look what item it's pointing to so I know what's going on.

It would be nice if the format would add comment-lines with a more human-friendly text that tells you what the ID actually stands for. This will help in resolving issues, and will save some time finding out what is what. For example:

ID: "e391b526-d0c5-439d-803e-17512eae6222" Hint: Allowed Controls Type: TreelistEx Value: | /* Rendering01 */ {A21617EF-4786-46AE-A8EC-C0A5A10A74F9} /* Rendering02 */ {0F3395F7-5B5D-40BF-958F-AB738B5BCD55} /* Rendering03 */ ...

Identical paths in different databases can't be serialized

When I use this configuration in unicorn this an error is thrown.

<predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
   <include database="core" path="/sitecore/system/Settings" />
   <include database="master" path="/sitecore/system/Settings" />
</predicate>

Throws this error

Multiple predicate include nodes had the same name 'Settings'. This is not allowed. Note that this can occur if you did not specify the name attribute and two include entries end in an item with the same name. Use the name attribute on the include tag to give a unique name.

Allow Exceeding Windows 250 character limit

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

Will not work if SerializationFolderPathMaxLength+ 8 + MaxItemNameLengthBeforeTruncation > 250

Desired Behavior

Ability to set even longer lengths or not be artificially restricted and then display error message when exception is thrown due to path length being an issue and to enable long path length on our OS.

Option name ideas:
Rainbow.Sfs.DisablePathLengthChecks or Rainbow.Sfs.AllowUnlimitedPathLength

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.

9.0.2 and 2.01

Question: Any thoughts on XMLField formatting using NewLineOnAttributes?

Hi Kam

Have you thought about using NewLineOnAttributes when formatting XML in XmlFieldFormatter? It should make it easier see what's changed when diff'ing commits.

Something like:

XDocument doc = XDocument.Parse(field.Value);

XmlWriterSettings settings = new XmlWriterSettings
{
    OmitXmlDeclaration = true,
    Indent = true,
    NewLineOnAttributes = true,
};
StringBuilder result = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(result, settings))
{
    doc.WriteTo(writer);
}

return result.ToString();

Which will output something like:

<r xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <d
    id="{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}"
    l="{0DAC6578-BC11-4B41-960A-E95F21A78D1F}">
    <p
      uid="{C4C0DC9E-3C95-45D2-BB02-EAA406B1C9A9}"
      key="main"
      md="/sitecore/layout/Placeholder Settings/Sites/Public/Home/main" />
    <r
      id="{EF37A63D-B9E1-4335-B4B1-762E4764EF5A}"
      ph="wrapper"
      uid="{6D364A99-1F9D-44F1-A849-8D8310DB24AE}" />
  </d>
</r>

EnforceVersionPresenceDisabler is missing from Sitecore 7

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
It fails with an exception when trying to use Unicorn on Sitecore 7.2

If the current behavior is a bug, please provide the steps to reproduce.
Install the latest version af Unicorn and Rainbow on Sitecore 7.2, serialize some items, and redeserialize them again, it will fail with an exception about EnforceVersionPresenceDisabler not being found in Sitecore.Kernel

What is the expected behavior?
It working :)

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.
Sitecore 7.2 (latest version), Rainbow 2.0.1, Unicorn 4.0.4

Item creation fails for Item with name starting with predicate

I have configuration as follows:

    <configuration name="Default Configuration">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
               <include name="Templates" database="master" path="/sitecore/templates/Algolia"/>
          </predicate>
    </configuration>

and with it I am getting an exception when create an item /sitecore/templates/AlgoliaTest.

Items with names not starting with Algolia works fine.

No trees contained the global path /sitecore/templates/AlgoliaTest

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.InvalidOperationException: No trees contained the global path /sitecore/templates/Algolia1

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[InvalidOperationException: No trees contained the global path /sitecore/templates/AlgoliaTest]
Rainbow.Storage.SerializationFileSystemDataStore.Save(IItemData item) +286
Unicorn.Data.DataProvider.UnicornDataProvider.SerializeItemIfIncluded(IItemData item, String triggerReason) +101
Unicorn.Data.DataProvider.UnicornDataProvider.AddVersion(ItemDefinition itemDefinition, VersionUri baseVersion, CallContext context) +812
Unicorn.Data.DataProvider.UnicornSqlServerDataProvider.AddVersion(ItemDefinition itemDefinition, VersionUri baseVersion, CallContext context) +172
Sitecore.Data.DataProviders.DataProvider.AddVersion(ItemDefinition item, VersionUri baseVersion, CallContext context, DataProviderCollection providers) +128
Sitecore.Data.DataSource.AddVersion(ID itemID, VersionUri baseVersion) +82
Sitecore.Nexus.Data.DataCommands.AddVersionCommand.Execute(Item item) +211
Sitecore.Data.Engines.EngineCommand2.Execute() +96 Sitecore.Data.Engines.DataEngine.AddVersion(Item item) +316 Sitecore.Data.Managers.ItemProvider.AddVersion(Item item, SecurityCheck securityCheck) +265 Sitecore.Data.Managers.DefaultItemManager.ExecuteAndReturnResult(String pipelineName, String pipelineDomain, Func1 pipelineArgsCreator, Func1 fallbackResult) +165 Sitecore.Data.Managers.DefaultItemManager.AddVersion(Item item, SecurityCheck securityCheck) +239 Sitecore.Nexus.Data.DataCommands.AddFromTemplateCommand.�(String �, Item �, ID �, ID �) +66 Sitecore.Data.Engines.DataCommands.AddFromTemplateCommand.DoExecute() +116 Sitecore.Buckets.Commands.AddFromTemplateCommand.DoExecute() +1302 Sitecore.Data.Engines.EngineCommand2.Execute() +96
Sitecore.Data.Managers.ItemProvider.AddFromTemplate(String itemName, ID templateId, Item destination, ID newId) +370
Sitecore.Data.Managers.<>c__DisplayClass3.b__2() +51
Sitecore.Data.Managers.DefaultItemManager.ExecuteAndReturnResult(String pipelineName, String pipelineDomain, Func1 pipelineArgsCreator, Func1 fallbackResult) +165
Sitecore.Data.Managers.ItemManager.AddFromTemplate(String itemName, ID templateId, Item destination, ID newItemId) +151
Sitecore.Data.Items.Item.Add(String name, TemplateID templateID, ID newItemID) +93
Sitecore.Data.Items.Item.Add(String name, TemplateID templateID) +154
Sitecore.Workflows.WorkflowContext.AddItem(String name, TemplateItem template, Item parent) +84
Sitecore.Shell.Framework.Commands.AddMaster.Add(ClientPipelineArgs args) +869

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +128
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146
Sitecore.Reflection.ReflectionUtil.InvokeMethod(MethodInfo method, Object[] parameters, Object obj) +89
Sitecore.Nexus.Pipelines.NexusPipelineApi.Resume(PipelineArgs args, Pipeline pipeline) +313
Sitecore.Web.UI.Sheer.ClientPage.ResumePipeline() +224
Sitecore.Web.UI.Sheer.ClientPage.OnPreRender(EventArgs e) +823
Sitecore.Shell.Applications.ContentManager.ContentEditorPage.OnPreRender(EventArgs e) +24
System.Web.UI.Control.PreRenderRecursiveInternal() +200
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +7738

DefaultDeserializer resets fields which are excluded in fieldFilter

Do you want to request a feature or report a bug?

Unknown

What is the current behavior?

I've excluded the Sitecore field "__Security" in the <fieldFilter> element of my Unicorn configuration. However, the field still gets reset during Sync when other fields of the same item have changed. This behaviour is somewhat described here, but it also states that the field should be "effectilvely ignored", which it is not. Therefore I'm not sure if this is a bug...

If the current behavior is a bug, please provide the steps to reproduce.

• Exclude a field in the <fieldFilter> element of your configuration (e.g. __Security)
• Change any value in another field in one of the serialized items. This way, the excluded field should be removed from the serialized item's file.
• Deserialize

What is the expected behavior?

Since I've excluded the field from Unicorn, I would expect this field to be completely ignored during serialization and deserialization.
I've added the necessary changes to the DefaultDeserializer including a unit test which illustrates this behaviour here: NaibafCH@9e6a352

Please mention your Sitecore version, Rainbow version, and if also using Unicorn, your version of that too.

Sitecore: 9.0 Update-2
Rainbow: 2.0.0
Unicorn: 4.0.3

Unicorn API for Reserialize

What is the API to call to perform unicorn reserialization??

$Name adding $ in TFS

Hi,
We are having branch template with $Name and with new version of Unicorn it get seriliazed as $Name.yaml and when try to add this in TFS it give error special character are allowed in TFS.

In Previous version of unicorn the $Name used to get serialized to %40Name.item and it is used to get added easily in TFS