[IntegTest] Onboard all plugins into automated integration tests for the distribution about opensearch-build

[MMakaria]

I Think in a glance these are missing: Perftop/ Job Schedule / Performance Analyzer/Common Utils and Trace Analytics

[sean-zheng-amazon]

kNN with security test cases have been added for both x64 and arm, please update.

[peterzhuamazon]

kNN with security test cases have been added for both x64 and arm, please update.

Updated. Thanks.

[peterzhuamazon]

So @bowenlan-amzn has informed me that IM opensearch has Security Enabled since this PR:
opensearch-project/index-management#93

He also mentions that IM Dashboards tests are now available, but flaky:

yarn run cypress run
yarn run cypress run --env security_enabled=true

[lezzago]

I have tested Alerting dashboards on OS arm64 manually.
I was able to successfully create a chime destination and a monitor.
I was able to have a trigger in the monitor get triggered and send chime messages to the chime destination.

[bbarani]

@lezzago Please make sure to update your automated testing to cover the above (manual testing) scenarios as well.

[dblock]

What’s the criteria of testing with security vs. without or both?

[peterzhuamazon]

Seems like Async Search WITH SEC is up. Need confirmation.

[peterzhuamazon]

Seems like SQL Workbench WITH SEC is up. Need confirmation.

[peterzhuamazon]

As of 2021/11/22, SQL Workbench has SEC, Notebook dashboards 1.1 and before has SEC, Reports dashboards has SEC, Gantt Charts has NOSEC + SEC.

So all the plugins in dashboards now have NOSEC + SEC tests.

As of now, everything can use 1.2 branch for 1.2.0 tests, only observability is using main for the test cases (backport later).

Dashboards notebooks, trace analytics and observability all merged into trace analytics plugin and repo, will rename to observability after 1.2.0 release.

[peternied]

@bbarani is there a campaign associated with driving this work? Checking in here for security as we realized this is a gap we'd like to address. Created issue in our repo for tracking opensearch-project/security#1761

CC @davidlago

[cliu123]

Security plugin will provide manual sign-off: opensearch-project/security#1499 (comment).

[peterzhuamazon]

Added notificationsDashboards now.

[lezzago]

Please note that this issue is slated to be resolved in 2.1 as this is an enhancement to the current integration tests that have already onboarded for the notification plugin.

It might be better to remove this notification issue from this github issue as this current issue revolves around onboarding automated integration tests for the distribution, which Notification plugin already has. The notification github issue mentioned here is just a further enhancement of the existing integration tests.