Long path names when using a Web-root-relative path causes errors about unicorn

[kamsar]

What version? Thought I had fixed something like this before. Could be wrong of course.

On Nov 20, 2014, at 2:17 AM, Martijn Bos [email protected] wrote:

When I use a web-root-relative path in the seializationProvider an error occurs when the path name is longer than x. 'path is not under root'

The serializationProvider looks like this

Item path can look like this
/sitecore/templates/MyTemplates/Base templates/Base navigation items folder/New template with a long name/New long section name/New field name with a very very very long name

The full error looks like this

[Exception: path is not under the root]
Sitecore.Data.Serialization.PathUtils.GetShortPath(String path) +168
Sitecore.Data.Serialization.PathUtils.MapItemPath(String itemPath, String root) +383
Unicorn.Serialization.Sitecore.SerializationPathUtility.GetSerializedReferencePath(String rootDirectory, ISourceItem sourceItem) +93
Unicorn.Serialization.Sitecore.SitecoreSerializationProvider.SerializeItem(ISourceItem item) +168
Unicorn.UnicornDataProvider.SerializeItemIfIncluded(ItemDefinition itemDefinition, String triggerReason) +126
Unicorn.UnicornSqlServerDataProvider.CreateItem(ID itemId, String itemName, ID templateId, ItemDefinition parent, CallContext context) +238
Sitecore.Data.DataProviders.DataProvider.CreateItem(ID itemID, String itemName, ID templateID, ItemDefinition parent, CallContext context, DataProviderCollection providers) +163
Sitecore.Nexus.Data.DataCommands.CreateItemCommand.Execute(ID itemId, String itemName, ID templateId, Item destination) +578
Sitecore.Data.Engines.DataCommands.CreateItemCommand.DoExecute() +108
Sitecore.Data.Engines.EngineCommand2.Execute() +122 Sitecore.Data.Engines.DataEngine.CreateItem(String itemName, Item destination, ID templateId, ID newId) +98 Sitecore.Data.Managers.ItemProvider.CreateItem(String itemName, Item destination, ID templateId, ID newId, SecurityCheck securityCheck) +363 Sitecore.Data.Managers.ItemManager.CreateItem(String itemName, Item destination, ID templateId, ID newId, SecurityCheck securityCheck) +215 Sitecore.Data.Managers.ItemManager.CreateItem(String itemName, Item destination, ID templateId, ID newId) +169 Sitecore.Nexus.Data.DataCommands.AddFromTemplateCommand.�(String �, Item �, ID �, ID �) +12 Sitecore.Data.Engines.DataCommands.AddFromTemplateCommand.DoExecute() +114 Sitecore.Data.Engines.EngineCommand2.Execute() +122
Sitecore.Data.Engines.DataEngine.AddFromTemplate(String itemName, ID templateId, Item destination, ID newId) +102
Sitecore.Data.Managers.ItemProvider.AddFromTemplate(String itemName, ID templateId, Item destination, ID newId) +364
Sitecore.Data.Managers.ItemManager.AddFromTemplate(String itemName, ID templateId, Item destination, ID newItemId) +204
Sitecore.Data.Managers.ItemManager.AddFromTemplate(String itemName, ID templateId, Item destination) +287
Sitecore.Data.Items.Item.Add(String name, TemplateID templateID) +146
Sitecore.Data.Items.TemplateSectionItem.AddField(String fieldName) +266
Sitecore.Shell.Applications.Templates.TemplateBuilder.TemplateBuilderForm.SaveField(TemplateSectionItem sectionItem, FieldDefinition field) +213
Sitecore.Shell.Applications.Templates.TemplateBuilder.TemplateBuilderForm.SaveSection(TemplateItem templateItem, SectionDefinition section) +561
Sitecore.Shell.Applications.Templates.TemplateBuilder.TemplateBuilderForm.SaveTemplate() +773
Sitecore.Shell.Applications.Templates.TemplateBuilder.TemplateBuilderForm.Save(ClientPipelineArgs args) +1183

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +76
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +211
System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) +35
Sitecore.Nexus.Pipelines.NexusPipelineApi.Resume(PipelineArgs args, Pipeline pipeline) +398
Sitecore.Pipelines.Pipeline.Start(PipelineArgs args, Boolean atomic) +327
Sitecore.Web.UI.Sheer.ClientPage.RunPipelines() +271
Sitecore.Web.UI.Sheer.ClientPage.OnPreRender(EventArgs e) +547
System.Web.UI.Control.PreRenderRecursiveInternal() +113
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4297
—
Reply to this email directly or view it on GitHub.

[Bosdongen]

Currently using 2.0.4

[akshaysura]

The current way the serialization works wont help. Its windows which is restricting the path length. You will have the same issue is you did Native Sitecore Serialization.

[kamsar]

This is the dirty little culprit in Sitecore.Data.Serialization.PathUtils:

 public static string GetShortPath(string path)
 {
   if (!path.StartsWith(PathUtils.Root, StringComparison.InvariantCultureIgnoreCase))
   {
    throw new Exception("path is not under the root");
   }
   return Path.Combine(PathUtils.Root, PathUtils.GetHash(path.Substring(PathUtils.Root.Length)));
 }

The method gleefully ignores any custom root passed, even though the parent MapItemPath() does respect the root. So if the path goes overlength, and the path is outside the official serialization root, it will explode.

You may be able to work around the issue by changing the SerializationFolder setting in web.config:

  <setting name="SerializationFolder" value="$(dataFolder)/serialization"/>

If this path is repointed to your out of webroot folder it should, hypothetically, fix the problem. I don't think it supports web root relative paths that leave the web root though like Unicorn does.

The eventual fix would be to replicate the PathUtils stuff that this does in Unicorn itself, and remove the root check in the above method.

[kamsar]

I am strongly thinking of doing a new serialization provider for the next version of Unicorn which will use an index file and a block of guid-named files to avoid all these very messy pathing and foldering issues that the regular Sitecore serialization introduces. Perhaps also a less verbose serialization format that focuses on essential data interchange (e.g. no 'last updated', 'revision', 'updated by' at all), which would reduce repository data size and reflect that fact that really the last modifier was always someone in git via unicorn.

Thoughts?

[AndersRasmussen]

@kamsar I think it would be a very helpful provider. We have some paths we simply can't serialize at the moment, which is a problem for us.
The lesser verbose serialization format should be an opt-in, because unicorn also can be used in not development environments, where changes are merged in to git occasionally and not by the modifier itself.

[DiegoSSJ]

Thanks guys for all the feedback, we just ran into this problem as well, while serializing core:/sitecore/client/Application

Now it might be our turn to share some ideas!

We have apparently solved the problem by changing the Sitecore setting SerializationFolderPathMaxLength:

We changed the value from 90 to 20. The longest item in the core database is:
"\sitecore\client\Applications\ExperienceAnalytics\Common\System\Texts\ErrorMessages\In the COMPONENT_NAME COMPONENT_TYPE multiple keys and multiple metrics are not supported Contact your system administrator"

And we didn't get that item serialized from default when that option was set to 90, but setting it to 20 did the trick.And that's including our own path to the serialization folder, which is neither long nor short, just average. So one could possibly serialize longer than that if one would make a link from say c:\s to the actual map, or do the serialization there.

Also as a side dev we have now the SQL to lookup items with longest path in database, if anyone is interested. It's just some google away, but it's nice when someone already has done the minutes long work.

OBS: We didn't seem to get the error mentioned by this issue, but we did not get the item serialized, neither by unicorn nor Sitecore.

[DiegoSSJ]

Also worth mentioning is that one can use a joint link to reduce the length of the path to the real serialization folder.

See http://stackoverflow.com/questions/1880321/why-does-the-260-character-path-length-limit-exist-in-windows

"

One way to cope with the path limit is to shorten path entries with symbolic links.

For example:

create a C:\p directory to keep short links to long paths
mklink /J C:\p\foo C:\Some\Crazy\Long\Path\foo
add C:\p\foo to your path instead of the long path

"

[kamsar]

Hi all,

It's pretty trivial to organize items by ID (e.g. current prototypes do $root\D\DEA5-38585-35435...)

However I'm concerned that it may make readability much harder - you can't really visualize what's in your serialization store if all you have is a block of GUIDs. What do you guys think?

One possible solution I've thought about is some sort of tree-reader application that is a generated tree-view of your serialized items where you can delete trees, etc.

[sitecorepm]

Hi all,

I ran into the same error trying to set my root path to "~/../data/serialization/unicorn" which IS under the path set with the SerializationFolder setting but because of the "/../" doesn't technically match the value (which is the fully resolved path).

So this raises the errror, "path is not under the root", as @kamsar points out above.

The solution for me was to replace "~/../data" with the sitecore config variable $(dataFolder) when setting my unicorn rootPath. Like this:

<serializationProvider type="Unicorn.Serialization.Sitecore.Fiat.FiatSitecoreSerializationProvider, Unicorn" 
rootPath="$(dataFolder)\serialization\unicorn" singleInstance="true" />

which then resolves to the full path and thus avoids the issue.

If you want to serialize your items elsewhere, then you can combine this solution with "joint link" method above. In my case, the "unicorn" folder is a joint link to a different location.

[kamsar]

Nice solution, I'd forgotten that Sitecore would do variable expansion everywhere in the config. Unicorn 3 will have a pathing provider that essentially snips overlength trees like Sitecore does, but the loopbacks are named by parent item ID and contain the complete child tree. Multiple loopbacks are allowed.

Hopefully this walks the awkward line between human readability - so merge tools aren't unintelligible - and being able to store deep trees.

U3 pathing also is not a complete tree. Only the relative path from the include root is stored. So for example if you have <include path="/sitecore/content/some/very/long/root/path/of stuff" /> in the predicate, and your serialization root was c:\foo your root item would be c:\foo\of stuff\of stuff .yml. This can be disambiguated with the new name attribute on the include, if you want to name the roots.

[sitecorepm]

Hey @kamsar - Will U3 handle ridiculously long item names? WFFM is usually the culprit here as it creates underlying form field items with names that are the user entered field label... such as "First Name", "Last Name" etc.. that's all fine until you get users writing a paragraph description for a field (which we have seen several instances of). This breaks the "path too long" rule no matter what because the item name itself is longer than 260. It also breaks Sitecore's built-in serialization.

[kamsar]

That's a good point. I think we can probably support that (by forcing item names that are too long to be {id}.yml). I'll add it to my stack of TODO :)

[kamsar]

Obscenely long filenames are now supported in Rainbow (by truncating the filename stored on disk):
SitecoreUnicorn/Rainbow@0a3a483
SitecoreUnicorn/Rainbow@75d587c

And yes, it has test coverage, and items that truncate to the same name are supported :)

[sitecorepm]

Nice!

[SimonRoodt]

You can point your dataFolder to a place inside or outside your webRoot. If you have it outside your webRoot you need to specify a physical path. Sitecore doesn't seem to support web relative paths pointing outside your webroot (eg. ~/../../data). Provided you are using the default Sitecore serialization folder (ie $(dataFolder)/Serialization), Sitecore has implemented a mechanism to shorten serialized paths to avoid the Windows 248 character limit. However, other serialization mechanism may not support this shortening, even though they may support web relative paths.

[kamsar]

Sitecore's short path mechanism is pretty buggy, hence why Unicorn 3 abandons it entirely. Unicorn provides the relative path support, even in v2 - the bugs are in the Sitecore PathUtils class.

[SimonRoodt]

thanks kam. so I guess the best guess is to use unicorn 3 then instead of
leveraging sitecore provider?
On 08 Sep 2015 3:47 PM, "Kam Figy" [email protected] wrote:

Sitecore's short path mechanism is pretty buggy, hence why Unicorn 3
abandons it entirely. Unicorn provides the relative path support, even in
v2 - the bugs are in the Sitecore PathUtils class.

—
Reply to this email directly or view it on GitHub
#34 (comment).

[kamsar]

The sitecore provider is fine unless you want to move paths around (or have really long item names, etc). But Unicorn 3's is nicer and can be extended or replaced.

This post goes into more detail.