fatihsahin / test-flask Goto Github PK

A visual studio extension can be implemented to mark methods with Playback functionality on VS editor. Marked methods with proper identifier can be persisted to a xml file and weaver can rely on this file to weave methods

When a response is being updated for a invocation, user can declare cascading options. For example user can trigger the update for all responses with matching depth, sibling, request or signature inside the step, with matching request or signature inside a scenario or whole project with a single update action.

Upgrade TestFlask to use new Fody version. New Fody licence is asking package consumers to be a patreon. TestFlask can contain its own plugin to Ms Build and therefore there would be no need for Fody. Worth considering.

As invocations are cleared before hand on initial service root depth, cross service cannot find matching invocation to replay in intellirecord mode even if the invocation was marked as replayable. Consider an alternative solution to clearing all invocations. Each service may manage its ow invocations inside a step, a service key should be introduced in configuration for services

For generic methods, we cannot define a generic attribute. It is not allowed in C#. Therefore it makes sense to declare an identifier like below with an object type for generic args.

public class GetGenericReqIdentifier : IRequestIdentifier<object, FooRequest>
{
	public string ResolveDisplayInfo(object arg0, FooRequest arg1)
	{
		return "ooh";
	}

	public string ResolveIdentifierKey(object arg0, FooRequest arg1)
	{
		return "whaaat";
	}
}

However, testFlask weaver generates something like

[Playback(typeof(GetGenericReqIdentifier), null)]
public TRes GetSomeGeneric<TReq, TRes>(TReq req, FooRequest fooReq) where TRes : new()
{
	FuncPlayer<TReq, FooRequest, TRes> funcPlayer = new FuncPlayer<TReq, FooRequest, TRes>
("TRes AssemblyToProcess.SomeClient`1::GetSomeGeneric(TReq,AssemblyToProcess.FooRequest)", 
		new GetGenericReqIdentifier(), null);
	//the rest of body
}

which generates a compiler error that is Argument 2: cannot convert from 'AssemblyToProcess.GetGenericReqIdentifier' to 'TestFlask.Aspects.Identifiers.IRequestIdentifier<TReq, AssemblyToProcess.FooRequest>'.

TestFlask must support generic identifier implementations

While playing a current step, current played invocation tree can be gathered just like in record mode. This tree can be used to log or report a difference between recorded tree and replayed one. Also unmatched invocations can be marked to alert changes in play

A CLI tool can be written to run a console app to trigger recorded steps by actually calling TestFlask ready backend service host to support regression testing automation. Console app can stream errors to ErrorOutputStream, so a defined TFS build task can plug in and fail the entire build process if any assertions fail

Introduce a new testflask mode like IntelliRecord/HybridRecord. When testFlask is set to this mode, inner player will record unmatched invocations as new and call original record wrapper, yet it will reuse recorded responses for matching invocations. Therefore it will be possible to record a diff patch to reduce the cost of setting up integration environments for re-recording. Only newly introduced integration nodes need to be online.

auto generated unit test methods can have summary doc sections feeded from scenario and step descriptions

Currently cross site raw requests are not persisted and therefore it is not possible to use variable replacement processing. Consider a solution.

TestFlask is now mature enough for a 1.0.0 version package release to nuget.org.

This is a very early brainstorming. Java services can be manipulated using maybe Javassist and a maven task plugged in for javassist

in order to fully deserialize recorded request and response objects, type names must be included in json strings. This way we can support reinstantiating objects with inherited types

When an invocation is requested from TestFlask API, currently it uses invocation instance hash code to fetch single invocation. Invocation instance hash code is a composite hash code to define a invocation instance uniquely inside the whole recorded call tree. However, different invocation matching strategies can be used such as using signature hash code, signature+request identifier hash code, and depth hash code. That way TestFlask could return same response for methods with same signature, same signature with same request identifier or same signature+request identifier on same depth level. This way api could return another invocation response and it may help regression tests to be independent from call stack tree changes. Strategy can be set on many levels including on project level, scenario level and step level. Lower levels would be dominant.

testFlask should be able to weave generic methods such as
public TRes DoSomething<TReq1, TReq2, TRes>(TReq1 req1, TReq2 req2, int someOtherParam) { }

HttpContext.Current is blocking migration to .net core. It also disables support for async call interceptions. I will use a static Dictionary and keep it simple. In async calls they will be directly appended to root call. In sync calls to distinguish initial invocation from childs or siblings I plan to mark initial invocation with sth like [Playback(IsRoot: true)]. A timer thread with some idle time configuration may clean up static dictionary after a recording session is complete and override the scenario if IsRoot:true invocation is recorded again or replayed.

testflask shoud be able to weave static methods

Create a POC branch for weaving async functions that returns Task or Task<T>. TestFlask context (such as depth, invocation index or hierarchy) can be synced with continuation tasks that are appended inside player. HttpContext may or may not be available inside thread pool. An alternative solution can be necessary. Also consider what can be done with async void methods

Clone scenario implementation inside StepController must also clone scenario variables.

An invocation totally declared from scratch or cloned from other step invocation trees can be patched to an existing invocation tree to make steps more flexible to react to underlying service version/implementation changes. UI Views can be implemented in test-flask-web repo, or a new rich client (like WPF maybe?) can be implemented for specific patching/cascading variations.

Users can define variables on project, scenario and step levels to replace any part of their recorded root request to be dynamically replaced with variable values. Moustache style {{variableName}} can be preffered. By altering variable values, request can be dynamically modified before any assertion or playback. This could help to trigger some secure service or apis with changing token or user credentials or with custom soap or http headers, idempotency keys etc.

Update libs to target .net standard latest version

Scenario cloning is currently not implemented. A clone should update stepno and scenariono containing fields in whole scenario object tree as well as variables to be cloned to the new scenario

Some variables can have a basic C# snippet to resolve the value in runtime. While declaring a variable, we can use the value field to store expression and a IsEvaluated flag to determine whether this is a static value or must be evaluated. Probably a very basic C# compiler would work and there is no need to support advanced syntax like LINQ at the beginning. It would be great if expression like this works => System.DateTime.Now.ToString("dd.MM.yyyy")

A side TestFlask project potentially could auto generate unit tests to trigger each scenario as a seperate test method. Generator can use root invocation request object and method signature to set up the test. It could also use expected assertion object to truly assert the output. It could help increasing the test code coverage.

Raw http requests that are kept inside invocations can be manipulated and parameterized dynamically during recording if a variable that can be applied is already declared and have a matching generator regex pattern.

Scenarios can have one or more labels to categorize the scenario trair explicitly. Unit test generator tool or other complementary features can rely on these labels to use the scenario or not in their specific context