activeviam / mac Goto Github PK

Some of the provided dashboards/tabs/widgets provided with the MAC project have an unclear name and/or an unclear description.

A review pass should be done on all of these.

After the initial load, statistic folders inserted in the application statistics directory do not trigger the parsing of their statistic files.

This may come from the fact that in the WatcherService an event is fired for the added directory, but not the files it contains.

If you try to compile the SpringBoot JAR and start MAC, it fails as it cannot load the JSON files into the Content Server.
Indeed the code is set to properly pick the files contained inside the JAR (BOOT-INF/classes).
This needs to be fixed as this is not usable to always rely on running it inside the IDE.

I have prepared a fix in the branch 5.10-FIX-START-JAR.

ActivePivot is capable of compressing chunks, creating single-value chunks, offset chunks, etc.
This information can be viewed in MAC. Yet, there is no measure for the saved space.
Intuitively, we could compute for each chunk a measure being chunk_size / chunk_visible_size, giving a ratio of the saved memory.

With AP 5.10 introducing a change in the structure of the memory statistics, it would probably useful to have tests run on pre-generated statistic files coming from different AP versions in order to test compatibility with all supported versions.

Proper versioning in the statistic parsing code would also lead to cleaner code in the long run, as the current solution could easily get out of hand after a few updates to the statistic structure.

Something I started doing a while ago and spoke to OPE about, but haven't had chance to revisit. I've decided to create an Issue to get my thoughts down so that they are not lost.

As per the title, consider adding additional metadata.

For example when I browse the cube and create a table I can see the memory impact of a store of various fields, but I don't have much information around the usage of the store of fields. It would be good to know:

• the size (number of rows) in the store - I can then create a dashboard to see how the memory usage grows with the number of records loaded in the store
• a field being dictionarised or not - you can hack around and figure this out whether it's in the cube or not, but would be nice to know from a datastore level. The idea being that I could create a table showing the memory of the system vs datastore fields and whether they are dictionarised or not. For any none cube levels, I may than to explicitly dictionarise that field and reduce memory
• Cardinality of a field - eg could be used when analysing whether it's a suitable field for a secondary index or inclusion to the aggregate providers

I will add to this if I get any more ideas.

MAC should be updated to support memory statistics of AP 5.10 application.

With the recent changes to MAC, as well as the documentation of its measures and hierarchies, the ManagerDescriptionConfiguration class has become very busy. It could maybe gain from a rearrangement to have a structure like the one of the sandbox (where the measure definitions, hierarchies definition, and manager definition are all split in their own class).

The way constants are defined could maybe be revisited, to maybe have a pivot counterpart of the DatastoreConstants class.

In addition to the dump name, the epoch of the store/cube being read must be part of the key for every chunk.
Because it is possible to export several epochs in a single dump, this must be supported.

Exporting many epochs allow to compare the impact to the memory consumption of a given change to data.

Currently, the bookmark documentation is specified within the bookmark, as a string assigned to a content key in the file. Because it is standard JSON, multi line strings do not exist and the whole content is compressed into one line.

As a result, updating the documentation from a bookmark can be tedious as it involves either editing manually the single line string, or opening mac, using ActiveUI's text editor, and reexporting the bookmark with the changes.

Two possible solutions:

• Add a compilation step that would run a custom preprocessor on the bookmark files to inject content into them before running the application
• Add a custom ActiveUI plugin that would fetch documentation files from the content server

(@leguenv: already enough progress on the first solution to create a PR)

The "Oversized chunks" tab is currently unclear and does not bring up the relevant information to the user

Current view :

The graph is not clear and the neither is the content of the left table

The content of that tab should be clearer in order to be usabel

New table mockup/suggestion: