dconnett-splunk / splunk-add-on-for-dynatrace

Setup of developer env

Note: Must install docker desktop, vscode or pycharm pro optional

Note2: Appinspect requires libmagic verify this has been installed correctly each time a new workstation/vm is used https://dev.splunk.com/enterprise/docs/releaseapps/appinspect/splunkappinspectclitool/installappinspect

git clone [email protected]:splunk/<repo slug>.git
cd <repo dir>
git submodule update --init --recursive

#setup python venv must be 3.7
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements_dev.txt
pip install https://download.splunk.com/misc/appinspect/splunk-appinspect-latest.tar.gz

Test

Using docker

pytest

Using external Splunk instance with Eventgen and app pre-installed

pytest --splunk-type=external --splunk-host=something --splunk-user=foo --splunk-password=something

splunk-add-on-for-dynatrace's Issues

SSL/TLS issue: cert.pem “File name too long” ----BEGIN CERTIFICATE...."

Describe the bug
SSL/TLS issue: cert.pem “File name too long” ----BEGIN CERTIFICATE...."

Expected behavior
A clear and concise description of what you expected to happen.

Too much metrics with the same timestamp

Some of the customer report that after some time add-on stop to add metrics in Splunk, this can be due to the same timestamp for a lot of metrics, according to this message from customer:

From Customer: I think the issue will center around this error in internal logs.
01-22-2024 06:54:02.724 -0600 WARN AggregatorMiningProcessor [5267 merging] - Too many events (100K) with the same timestamp: incrementing timestamps 1 second(s) into the future to insure retrievability - data_source="dynatrace_timeseries_metrics://dynatrace_time_series_metrics", data_host="splk.customer.com", data_sourcetype="dynatrace:metrics"
01-22-2024 06:33:21.543 -0600 WARN DateParserVerbose [5267 merging] - The same timestamp has been used for 100K consecutive times. If more than 200K events have the same timestamp, not all events may be retrieveable. Context: source=dynatrace_timeseries_metrics://dynatrace_time_series_metrics|host=splk.customer.com.com|dynatrace:metrics|\n 1 similar messages suppressed. First occurred at: Mon Jan 22 06:24:01 2024

Wrong parameter being passed to entities endpoint and metrics endpoint

Describe the bug
time is being passed as the parameter for entities and metrics endpoints. The field for these endpoints is inconsistent with other parts of the Dynatrace API and needs from instead of time.

The Dynatrace API defaults to 2 hours due to not recognizing the time parameter.

See documentation here:
https://docs.dynatrace.com/docs/dynatrace-api/environment-api/metric-v2/get-data-points#parameters
https://docs.dynatrace.com/docs/dynatrace-api/environment-api/entity-v2/get-entities-list#parameters

To Reproduce
Steps to reproduce the behavior:

1. Create an input for entities or metrics.
2. Select any time interval other than two hours.
3. Two hours of data is returned, regardless of the interval.

Expected behavior
The interval should reflect what the user has set.

Desktop (please complete the following information):
All platforms are affected.

Additional context
This is the code that is used calculate and rewrite the fields per endpoint:

splunk-add-on-for-dynatrace/package/bin/util.py

Lines 340 to 368 in 7f0b089

	def find_format_key(value: str) -> Optional[str]:
	"""Find the key to be formatted from a given string value."""
	format_keys = [tup[1] for tup in Formatter().parse(value) if tup[1] is not None]
	return format_keys[0] if format_keys else None
	def get_formatted_key_value_pair(key: str, value: str, params: Params) -> Tuple[str, Any]:
	"""Return the key-value pair after formatting."""
	replaced_key = find_format_key(value)
	if replaced_key and replaced_key in params:
	formatted_value = value.format(**{replaced_key: params[replaced_key]})
	return key, formatted_value
	return key, value
	def format_url_and_pop_path_params(endpoint: Endpoint, url: URL, params: Params) -> Tuple[URL, Params]:
	"""Format the URL and pop the URL Path parameter from the params dictionary."""
	# print('format_url_and_pop_path_params: {}'.format(endpoint)
	# + ' url: {}'.format(url)
	# + ' params: {}'.format(params))
	if endpoint.url_path_param:
	endpoint_path_key = endpoint.url_path_param
	if endpoint_path_key in params:
	formatted_url = URL(url.format(**{endpoint_path_key: params[endpoint_path_key]}))
	new_params = Params({k: v for k, v in params.items() if k != endpoint_path_key})
	# print('format_url_and_pop_path_params: {}'.format(formatted_url)
	# + ' new_params: {}'.format(new_params))
	return formatted_url, new_params
	return url, params

The field mappings per endpoint are defined here:

splunk-add-on-for-dynatrace/package/bin/util.py

Lines 58 to 160 in 7f0b089

	class Endpoint(Enum):
	METRICS = \
	EndpointInfo(
	URL('/api/v2/metrics'),
	ResponseSelector('metrics'),
	Params({'writtenSince': '{time}',
	'fields': 'unit,aggregationTypes'}),
	None)
	METRICS_QUERY = \
	EndpointInfo(
	URL('/api/v2/metrics/query'),
	ResponseSelector('result'),
	Params({'metricSelector': '{metricSelector}'}),
	None)
	METRIC_DESCRIPTORS = \
	EndpointInfo(
	URL('/api/v2/metrics/{metricKey}'),
	ResponseSelector('metricId'),
	None,
	PathParam('metricKey'))
	ENTITIES = \
	EndpointInfo(
	URL('/api/v2/entities'),
	ResponseSelector('entities'),
	Params({'entitySelector': 'type(\"{entitySelector}\")'}),
	None,
	["HOST", "PROCESS_GROUP_INSTANCE", "PROCESS_GROUP", "APPLICATION", "SERVICE", "SYNTHETIC_TEST",
	"SYNTHETIC_TEST_STEP"])
	ENTITY = \
	EndpointInfo(
	URL('/api/v2/entities/{entityId}'),
	ResponseSelector("entityId"),
	None,
	PathParam('entityId'))
	PROBLEM = \
	EndpointInfo(
	URL('/api/v2/problems/{problemId}'),
	ResponseSelector('problemId'),
	None,
	PathParam('problemId'))
	PROBLEMS = \
	EndpointInfo(
	URL('/api/v2/problems'),
	ResponseSelector('problems'),
	None,
	None)
	EVENTS = \
	EndpointInfo(
	URL('/api/v2/events'),
	ResponseSelector('events'),
	None,
	None)
	SYNTHETIC_LOCATIONS = \
	EndpointInfo(
	URL('/api/v2/synthetic/locations'),
	ResponseSelector('locations'),
	None,
	None)
	SYNTHETIC_TESTS_ON_DEMAND = \
	EndpointInfo(
	URL('/api/v2/synthetic/executions'),
	ResponseSelector('executions'),
	Params({'schedulingFrom': '{time}'}),
	None)
	SYNTHETIC_TEST_ON_DEMAND = \
	EndpointInfo(
	URL('/api/v2/synthetic/executions/{executionId}/fullReport'),
	ResponseSelector('entityId'),
	None,
	PathParam('executionId'))
	SYNTHETIC_MONITORS_HTTP = \
	EndpointInfo(
	URL('/api/v1/synthetic/monitors'),
	ResponseSelector('monitors'),
	None,
	None)
	SYNTHETIC_MONITOR_HTTP = \
	EndpointInfo(
	URL('/api/v1/synthetic/monitors/{entityId}'),
	ResponseSelector('entityId'),
	None,
	PathParam('entityId'))
	SYNTHETIC_MONITOR_HTTP_V2 = \
	EndpointInfo(
	URL('/api/v2/synthetic/execution/{monitorId}'),
	ResponseSelector('monitorId'),
	None,
	PathParam('monitorId'),
	["SUCCESS", "FAILED"])
	# This is a hack because SYNTHETIC_MONITORS_HTTP returns entityIds and others return monitorIds
	SYNTHETIC_MONITOR_ENTITY_V2 = \
	EndpointInfo(
	URL('/api/v2/synthetic/execution/{entityId}'),
	ResponseSelector('entityId'),
	None,
	PathParam('entityId'),
	["SUCCESS", "FAILED"])
	SYNTHETIC_TESTS_RESULTS = \
	EndpointInfo(
	URL('/api/v2/synthetic/tests/results'),
	ResponseSelector('results'),
	None,
	None)

Missing data from entities APIv2

Describe the bug
Entities API does not contain all the expected data. Reports are coming in that some entityIds are missing.

Additional context
We cannot reproduce this in dev or test environments. It seems to be related to the load the system is under.