Add report system about auraxium

[leonhard-s]

Report Contents (WIP)

Basic Character Overview

• Kills
• Deaths
• KDR
• "True" KDR
• Faction
• Server
• Most played class
• Last played profile
• Battle rank / ASP
• Last seen
• Outfit

Extended Character Overview

• ACC
• HSR
• Playtime by profile
• Vehicle KDR
• Infantry KDR
• Top 5 weapons and related stats
• "Siege level" (see Voidwell)
• Facility captures/defences
• Assists

Vehicle Overview

• (Stats broken up by vehicle)

Weapon Overview

• (Stats broken up by weapon)

[leonhard-s]

That is quite likely. The item types also contain dummy objects used for mounts, player homes and similar shenanigans. EverQuest has left its mark on the engine for sure.

[leonhard-s]

After a bit of spring cleaning that led to v0.2.0, I got around to adding a tentative API for the reports system to the feature/reports branch.

Here is a non-exhaustive rundown of the current WIP API:

Report Base Class

The Report base is a generic mapping of Ps2Object instances to arbitrary types. This allows type checkers to both validate the objects passed and allows defining external data classes or typed dictionaries as the return type:

@dataclasses.dataclass(frozen=True)
class OutfitStats:
  """Read-only data class for outfit statistics."""
  ...
  
class OutfitStatReport(Report[ps2.Outfit, OutfitStats]):
  """A hypothetical report generating `OutfitStats` instances for any `ps2.Outfit` passed."""
  ...

Creating Reports

The Report ABC has two abstract class methods that must be implemented to create a working report.

The first is the query() factory, which creates the auraxium.census.Query used to retrieve the required data:

auraxium/auraxium/reports/base.py

Lines 234 to 235 in 866fb95

	def query(cls, ids: Iterable[int], client: Client) -> Query:
	"""Return the query returning the raw dat for this report.

The other is convert(), which converts the returned payloads into whatever format the report generates (e.g. an instance of the OutfitStats data class in the previous example).

auraxium/auraxium/reports/base.py

Lines 180 to 181 in 866fb95

	def convert(cls, data: Iterable[CensusData]) -> Dict[int, _T]:
	"""Convert the census response into a list of report outputs.

I implemented the vehicle statistics example as a test as this uses a complex query, the example script can be found in the branch's examples:

auraxium/examples/vehicle_damage_report.py

Lines 32 to 155 in 866fb95

	class VehicleDamageOverview(reports.Report[ps2.Item, _ReturnType]):
	"""A custom report to illustrate the Reports interface.
	This report takes in :class:`auraxium.ps2.Item` instances or
	integers and generates a report for each entry passed.
	"""
	# NOTE: The abstract query() and convert() methods are class methods to
	# allow access to other utility functions or classes defined within the
	# Report body.
	# For this example, they may as well be static methods as this capability
	# is not used.
	@classmethod
	def query(cls, ids: Iterable[int], client: auraxium.Client) -> census.Query:
	"""Report query factory.
	This method is responsible for returning an
	:class:`auraxium.census.Query` instance that will retrieve the
	data needed to generate the report.
	"""
	# Value discarded to avoid "unused parameter" warnings
	_ = cls
	query = super().query(ids, client=client)
	# NOTE: The following query generation block is rather complex as it
	# has to work around some API limitations the details of which are not
	# within the scope of this example.
	# For more information, refer to the following Wiki section:
	# https://github.com/leonhard-s/auraxium/wiki/Census-API-Primer#join-limitations
	query.data.collection = 'item'
	query.add_term('item_id', ','.join((str(i) for i in ids)))
	query.show('item_id', 'name.en')
	def get_fire_mode_join(query: census.Query) -> census.JoinedQuery:
	join = query.create_join('item_to_weapon')
	return (
	join.create_join('weapon_to_fire_group', fire_group_index=0)
	.set_fields('weapon_id')
	.create_join('fire_group_to_fire_mode', fire_mode_index=0)
	.set_fields('fire_group_id', 'fire_group_id'))
	# Main payload + projectile join
	(get_fire_mode_join(query)
	.create_join('fire_mode_2')
	.set_fields('fire_mode_id')
	.show('fire_mode_id', 'max_damage',
	'max_damage_ind', 'max_damage_ind_radius',
	'min_damage_ind', 'min_damage_ind_radius')
	.create_join('fire_mode_to_projectile')
	.set_fields('fire_mode_id', 'fire_mode_id')
	.create_join('projectile')
	.show('speed'))
	# Direct damage effect join
	(get_fire_mode_join(query)
	.create_join('fire_mode_2')
	.set_fields('fire_mode_id')
	.show('damage_direct_effect_id')
	.create_join('effect')
	.set_fields('damage_direct_effect_id', 'effect_id')
	.show('effect_id', 'target_type_id', 'resist_type_id')
	.create_join('resist_type')
	.set_fields('resist_type_id'))
	# # Indirect damage effect join
	(get_fire_mode_join(query)
	.create_join('fire_mode_2')
	.set_fields('fire_mode_id')
	.show('damage_indirect_effect_id')
	.create_join('effect')
	.set_fields('damage_indirect_effect_id', 'effect_id')
	.show('effect_id', 'target_type_id', 'resist_type_id')
	.create_join('resist_type')
	.set_fields('resist_type_id'))
	return query
	@classmethod
	def convert(cls, data: Iterable[CensusData]) -> Dict[int, _ReturnType]:
	"""Generate report instances from the census payloads.
	This method is the main converter factory. Here the payloads
	are broken up and used to populate the report's return type.
	"""
	_ = cls
	ret: Dict[int, _ReturnType] = {}
	item: Any
	for item in data:
	id_ = int(str(item['item_id']))
	# Create shorthands for nested dictionaries
	fire_mode_2 = (
	item['item_id_join_item_to_weapon']
	['weapon_id_join_weapon_to_fire_group']
	['fire_group_id_join_fire_group_to_fire_mode']
	['fire_mode_id_join_fire_mode_2'])
	projectile = (
	fire_mode_2['fire_mode_id_join_fire_mode_to_projectile']
	['projectile_id_join_projectile'])
	effect_direct = (
	fire_mode_2['damage_direct_effect_id_join_effect'])
	resist_direct = effect_direct['resist_type_id_join_resist_type']
	effect_indirect = (
	fire_mode_2['damage_indirect_effect_id_join_effect'])
	resist_indirect = effect_indirect[
	'resist_type_id_join_resist_type']
	# Construct return dictionary
	ret[id_] = {
	'speed': int(projectile['speed']),
	'damage': int(fire_mode_2['max_damage']),
	'damage_target_type': int(effect_direct['target_type_id']),
	'damage_resist_type': (
	int(effect_direct['resist_type_id']),
	str(resist_direct['description'])),
	'indirect_damage_max': int(fire_mode_2['max_damage_ind']),
	'indirect_damage_max_range':
	int(fire_mode_2['max_damage_ind_radius']),
	'indirect_damage_min': int(fire_mode_2['min_damage_ind']),
	'indirect_damage_in_range':
	int(fire_mode_2['min_damage_ind_radius']),
	'indirect_damage_target_type':
	(int(effect_indirect['target_type_id'])),
	'indirect_damage_resist_type':
	(int(effect_indirect['resist_type_id']),
	str(resist_indirect['description']))}
	return ret

Using Reports

Fortunately, using reports has a much simpler interface (that was the point, after all). There are currently two options, a class-based interface and a lazy, instance-based one.

Class-based interface

In the class-based interface, the user passes the elements to look up to the Report.gather() coroutine:

auraxium/auraxium/reports/base.py

Lines 200 to 203 in 866fb95

	@classmethod
	async def gather(cls, targets: Iterable[Union[int, _KT]],
	client: Client) -> Dict[Union[int, _KT], _T]:
	"""Return a mapping of `targets` to their report output.

Usage:

outfit_list: List[ps2.Outfit] = ...

results: Dict[ps2.Outfit, OutfitStats] = await OutfitStatReport.gather(outfit_list, client=auraxium.Client(...))

All elements are merged into a single query and fetched at once. This makes this a slow but efficient option for small to medium sized lists of elements.

Instance-based interface

In this variant, the Report is instantiated and used as an instance. Awaiting this instance behaves similar to the class-based variant, with all elements being fetched at once.

Alternatively, the user can asynchronously iterate over the Report instance. In this case, the elements passed are returned one-by-one, but the data required is retrieved in batches of 100 (default) elements.

auraxium/auraxium/reports/base.py

Lines 111 to 119 in 4422261

	def __init__(self, targets: Iterable[Union[int, _KT]],
	client: Client, batch_size: int = 100) -> None:
	"""Instantiate a report for the given targets.
	Instantiated reports can either be awaited to return the entire
	list of values at once, or can be asynchronously iterated over
	to batch process the given targets. The latter is generally
	preferrable for potentially very large input sets such as
	outfit members or character-wide statistics.

Usage:

outfit_list: List[ps2.Outfit] = ...

report = OutfitStatReport(outfit_list, client=auraxium.Client(...))

item: ps2.Outfit
data: OutfitStats
async for item, data in report:
  ...

This allows efficiently traversing input lists of any size that would otherwise produce unreasonably long query strings.

[leonhard-s]

Any feedback or comments on this interface would be much appreciated.

I will start implementing some of the built-in report types as per the previous comments to further test the interface in the coming days if no issues are found. ETA 1-2 weeks, hopefully. 👌

[leonhard-s]

Another notable example for the report system would be the single_character_by_id collection, which only makes sense in this context of a larger, optimised batch report.

This could even be a dynamic, user-subclassable system that can be used to further improve performance without having to leave the object model endpoints.

[EllyArd]

As I mentioned in #44, I was looking for relevant weapon stats, queried the datasheet, and the weapon stats are useless because of DBG API idiosyncrasies.

I can't grok the API enough to do the queries myself, but using the query the wiki page uses I can tell you relevant keys.

weapon on-paper comparison stats (ns-11a):

"move_modifier": "1",
"projectile_speed_override": "640",
"sprint_fire": "0",
"turn_modifier": "1",
"use_in_water": "0",
"zoom_default": "1",
"fire_refire_ms": "92", # represented as miliseconds per bullet
"fire_pellets_per_shot": "1",,
"reload_chamber_ms": "450", # reload + chamber = long reload
"reload_time_ms": "2000",
"max_damage": "143",
"max_damage_range": "10",
"min_damage": "125",
"min_damage_range": "65",
"shield_bypass_pct": "0", # Relevant for 2 NC weapons

recoil would need to be per firegroup (ADS vs not), but is:

"cof_override": "0",
"cof_pellet_spread": "0",
"cof_range": "50",
"cof_recoil": "0.1",
"cof_scalar": "1",
"cof_scalar_moving": "1",
"recoil_angle_max": "-17",
"recoil_angle_min": "-20",
"recoil_first_shot_modifier": "3",
"recoil_horizontal_max": "0.2",
"recoil_horizontal_min": "0.2",
"recoil_horizontal_tolerance": "0.6",
"recoil_increase": "0.1",
"recoil_increase_crouched": "0.1",
"recoil_magnitude_max": "0.22",
"recoil_magnitude_min": "0.22",
"recoil_max_total_magnitude": "0",
"recoil_recovery_acceleration": "1000",
"recoil_recovery_delay_ms": "92",
"recoil_recovery_rate": "18",
"recoil_shots_at_min_magnitude": "0",

My third request would be something that groups vehicle damage statistics with their resist type, like

"speed": "250",
"damage_radius": "2",
"damage_type": "Damage",
"damage": "700",
"damage_target_type": "2",
"damage_resist_type": "7",
"indirect_damage_max": "500",
"indirect_damage_max_range": "1",
"indirect_damage_min": "50",
"indirect_damage_min_range": "3",
"indirect_damage_target_type": "2",
"indirect_damage_resist_type": "6"

and, for bonus points, spitting out those resists by name along with resist-ID. This allows us to easily make tools that, say, check for which liberator noseguns do the most damage to a bastion hardpoint at a given range, including reload. Or what kills a mosquito faster, a deployed or undeployed Colossus cannon? (Undeployed, surprisingly. Different resists.)

These are obviously big asks, and I don't intend it to be all on you. I can't contribute right now, but I'm hoping someone else with more spare energy comes through and can help with this. Short term I will hack the queries together and can post those, but I don't know Auraxium enough to translate them yet.

[leonhard-s]

Thank you for the templates - I hope I can find some time to work on this later this month.

As for translating the queries - the auraxium.census module is what generates the URLs, and it's interface is mostly a carbon copy of the API's structure; every API command has a corresponding method, and so on.
Moving between that and the object model is another matter that is currently woefully undocumented, though there is an introduction on the (work-in-progress) docs/rewrite branch (it uses references to external files so building that branch with Sphinx may be needed).

[EllyArd]

Yeah, I think I understand the structure a bit better now that I've thought about it. I already knew it was probably due to their end and not yours.

I'm slowly trying to get the structure of the API for another project and once I do if I have time I'll start trying to implement some of these. I wish the game API wasn't quite so labrynthine, but I'm sure there are game engine reasons for it.

[leonhard-s]

Closing this issue as none of the WIP report systems were flexible enough to warrant the added complexity.

A manual way to create helper methods similar to those the original reports drafts has been documented on RTD and is both more performant and flexible than class-based reports.

Users looking to recreate a Reports-like class interface may create a dataclass and use a helper method as a class method factory.