Comments (12)
@avishnyakov - It seems unlikely, but give the -Quiet parameter a try, it will suppress progress output. You might also take a look at PoshRSJob from @proxb, it's a fantastic, more fully featured framework for parallelization over runspaces.
@vors, @TravisEz13 - any plans to open source the jobs-based-tool that you mentioned? Seems like another path that might help here.
Cheers!
from invoke-parallel.
Could be a lot of different things. Invoke-Parallel
code is relatively small. If you have time to investigate, to add debug output to different places and gather this data, it could be a great help to nail down the issue.
I seen you opened powercode/PSParallel#4
Are you looking for a good framework for the parallel execution in PS? :)
from invoke-parallel.
Hey mate, thank you for getting back so fast.
I did open similar issue as the same story is happening with PSParallell. Furthermore, PSParallel stuck even with the PS console. Works only with ISE. Going to explore and run under VSO build agents.
The story around that has with Azure provision. We tend to parallel tons of operations getting the best of Azure PowerShell + either PSParallel or this Invoke-Parallel project. A wrapper on top of "parallel" execution helps us to switch between actual parallel run implementation, so that with one setting we either run Invoke-Parallel or PSParallel.
I stumbled upon that stuck behavior with TFS Online build agents. Troubleshooting today and tomorrow. Could be agent settings, could be something related to non-unteractive mode, could be silent switch or just something else. Could even be our Azure or PS mess as we do tons of weird stuff witin every parallel session. Need to try on bare array processing :)
And yes, there is almost NOTHING on really awesome PS parallel execution framework. PSParallel and your stuff really rock, guys. I really don't want to go with jobs. Been there, kinda hard to implement + pass params + get the feedback + return the values. I can invest time and make it right yet your scripts do a better job.
from invoke-parallel.
Has something to do with calling Azure PowerShell command within parallel loop..
PSParallel works well on simple array, stuck on Azure PowerShell calls.
Checking Invoke-Parallel, gonna be interesting day.
from invoke-parallel.
Oh! What Azure PowerShell commands are hanging up? I've been using invoke-parallel at $work recently. Occasionally (once or twice) get issues auto-loading the module in the first item, but works without issue the majority of the time. Is there a specific command/module you're running into trouble with?
Cheers!
from invoke-parallel.
Oh, Invoke-Parallel does not seem to import functions, only vars and modules. That's why we switched to PSParallel and PSParallel now hangs up on Azure PowerShell calls. Such a mess,
from invoke-parallel.
Okay, added custom function import into the initial session state:
#If specified, add variables and modules/snapins to session state
$sessionstate = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$excludedFunctionNames = @(
"cd",
"cd\",
"cd..",
"Pause",
"oos",
"mkdir",
"help"
)
$customFunctions = get-command | Where-Object {
$_.Module -eq $null `
-and $_.Name.Contains(":") -eq $false `
-and $excludedFunctionNames.Contains($_.Name) -eq $false `
-and $_.CommandType -eq "Function"
}
foreach($customFunction in $customFunctions) {
if($customFunction.Name -eq "Invoke-Parallel"){
continue;
}
Write-Verbose "Importing function:[$($customFunction.Name)]"
$functionImpl = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $customFunction.Name, $customFunction.Definition
$sessionstate.Commands.Add($functionImpl)
}
That thing would import custom written functions from the current session into the new runspace. That's was kinda irritating issue initially that pushed me to PSParallel.
That's a temporary fix only, as I see that a proper option and built-in stuff needs to be added. I see that happening as an additional parameter + $StandardUserEnv variable needs to be enhanced with 'custom function bit'.
It would be really nice to see custom function import. My ad-hoc fix is only adding functions with a particular name convention (suits us), skips some built-in functions (just in case) and assumes the rest comes from the modules.
Testing under ISE, console and TFS agent stuff. Will keep you posted with the outcome. Hope that helps.
from invoke-parallel.
@avishnyakov Have you looked at PoshRSJob (https://github.com/proxb/PoshRSJob) to see if it does what you are looking to do? I am curious to see how it performs with your use case.
from invoke-parallel.
No, I haven't yet. Pushing and testing Invoke-Parallel at this very moment.
PoshRSJob is the next in the list, thanks to you.
from invoke-parallel.
Console / ISE works well.
TFS Build agent works well but somehow gives 'false exception' all the time in the $runspace.powershell.Streams.Error
VERBOSE: Exception calling "Invoke" with "1" argument(s): "Cannot index into a null array."
VERBOSE: System.Management.Automation.MethodInvocationException: Exception calling "Invoke" with "1" argument(s):
"Cannot index into a null array." ---> System.Management.Automation.RuntimeException: Cannot index into a null array.
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception
exception)
at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean
createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior
errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo
invocationInfo, Object[] args)
at System.Management.Automation.ScriptBlock.<>c__DisplayClassa.<InvokeWithPipe>b__8()
at System.Management.Automation.Runspaces.RunspaceBase.RunActionIfNoRunningPipelinesWithThreadCheck(Action action)
at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior
errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo
invocationInfo, Boolean propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDefine, Object[]
args)
at System.Management.Automation.ScriptBlock.DoInvoke(Object dollarUnder, Object input, Object[] args)
at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
--- End of inner exception stack trace ---
at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type
typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
VERBOSE: at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception
exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
VERBOSE:
from invoke-parallel.
Confirming that with custom function import all works as expected across the board.
The exception early seems to be our scripts, errors out only on the build controller and it's extremely hard to find the exact location due to mixture or script block invoking and Invoke-Parallel. Commenting out help.
Closing, thanks all for patience.
from invoke-parallel.
Ans our build controller happen to have the old Azure PowerShell module, like 1.3.x something. That steals everything. All sorted now, Invoke-Parallel rocks!
from invoke-parallel.
Related Issues (20)
- NullReferenceException HOT 1
- Physical Memory usage HOT 2
- Log file locks if multiple scripts invoke-parallel HOT 4
- Get-RunspaceData error: Collection was modified; enumeration operation may not execute HOT 1
- it is not working to get AD group member using Get-ADGroupMember HOT 1
- Get-RunspaceData : ERROR: Deployment failed HOT 2
- Invoke-Parallel: When launched within a function, does not carry param()-variables forward
- Invoke-Parallel Scriptblock [Enviornment]::GetEnvironmentVariable
- Using a shared & synchronized variable to hold a counter HOT 2
- How we can use Invoke-Parallel to fetch ADUser data? HOT 4
- Import-Modules in parallel when session profile loaded? HOT 1
- Issue with Get-WMIObject HOT 1
- Weird error running Invoke-Parallel
- Variable Creep HOT 3
- Adding item to external collection HOT 2
- Error A null key is not allowed in a hash literal HOT 1
- skips
- Invoke-Parallel skips processing some objects when throttle > 1 HOT 2
- Fast-Ping Sweep (recommendations or tricks) HOT 2
- $PWD inside the script block doesn't update after changing the current directory
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from invoke-parallel.