Author: Filip Vavera [email protected]
Propose API testing testing approach for NASA Sound API
Test areas:
- Standard behavior
- Consistency
- Search behavior (missing documentation)
- Limit is working
- API response structure (missing documentation)
- API key is working
- Edge cases behavior
- Security testing
- SSL certificate testing
- API key testing (what is his purpose?)
- Executable code in query
- SQL injection
- ...
- Load testing
- Response time (what is the specification?)
Testing if the expected inputs returns expected outputs.
- Check if two requests with identical query returns same result.
- Check if two requests with different API key (but same q and limit) return same result.
Test if search behaves as expected. There is no documentation about how the search should work so I cannot check if the result is correct. But it could be implemented for example that way that searched query is in sound's description, tags or name.
This tests should be automated when the correct search behavior is documented. Until then it will be just guessing.
Test if limit
argument is working as expected. Check the count
field and also number of entries in results
field.
- Test when
limit
is not present there are 10 results - Test when
limit
is 1 there is 1 result - Test when
limit
is 10 there are 10 results - Test when
limit
is 20 there are 20 results - Test when
limit
is 57 there are 57 results - Test when
limit
is 64 there are 64 results - Test when
limit
is 75 there are 64 results (there are just 64 sounds in the API)
Test if returned JSON has correct structure. There is no documentation in that field so I assume the example response has correct structure. Check if other responses has the same structure.
- Test if every
results
list entry contains fields:description
- String or nulldownload_url
- Stringduration
- Integerid
- Integerlast_modified
- DateTimelicense
- Stringstream_url
- Stringtag_list
- Stringtitle
- String
- Test if
download_url
has format"https://api.soundcloud.com/tracks/{id}/download"
- Test if
stream_url
has format"https://api.soundcloud.com/tracks/{id}/stream"
- Test if
last_modified
has format"%Y/%M/%d %H:%m:%s %Z"
Test if API key is working correctly.
- Test if request number 1001 in one hour returns correct response
429 Too many requests
when using normal API key. - Test if request number 31 in one hour returns correct response
429 Too many requests
when using DEMO_KEY from one IP address. - Test if request number 51 in one day returns correct response
429 Too many requests
when using DEMO_KEY from one IP address.
Difficult to test automatically (have to check to run the test just once per hour/day). Because of other tests there will not be exact number of requests available.
The headers can be tested for X-RateLimit-Limit
and X-RateLimit-Remaining
to check the limits but this will not check the actual behavior.
Test edge case, unexpected and extreme values.
Test search ability for edge cases.
- Test
q
param with no value - Test
q
param with various special characters - Test
q
param with quote chars in it - Test
q
param with eval (`) in it - Test
q
param with large values (10 000 characters)
Test limit ability for edge cases
- Test
limit
param with no value - Test
limit
param with value which is not number - Test
limit
param with 0 - Test
limit
param with negative integer - Test
limit
param with integer larger then max integer (2 147 483 647) - Test
limit
param with integer smaller then min integer (-2 147 483 648) - Test
limit
param with decimal number
Test API key feature for edge cases
- Test
api_key
param with no value - Test
api_key
param with randomly generated key - Test
api_key
param with large values (10 000 characters)
Test API for passing unexpected parameter
- Test parameter with string as value
- Test parameter with number as value
- Test parameter with no value
Test API for security
- Test
403 Forbidden
response is returned to request without API key (automated) - Test
403 Forbidden
response is returned to request with not valid API key (automated) - Test
429 Too many requests
is returned after 30 requests in hour when using DEMO_KEY from one IP address - Test
429 Too many requests
is returned after 50 requests in day when using DEMO_KEY from one IP address - Test
429 Too many requests
is returned after 1000 requests in hour when using proper API key - Test all HTTP methods are not accessible except the GET method (automated)
- Test HTTP headers for security issues
- Test 4xx and 5xx responses returns correct responses (not stack trace or other sensitive information)
- SQL injection - SQL query in
q
param
- Test API under stress conditions
- Difficult to test since there is limit 1000 requests per API key. Testing would require unlimited API key.
- I would test it with JMeter. Create bunch of small servers on AWS or Google Cloud Platform and run in them Docker image of JMeter slave server (for example hhcordero/docker-jmeter-server ). And then run lot of concurrent API requests from all of the servers at once and watch the server load. Then mark and compare with specification:
- Standard load (around CPU load at 75%)
- Critical load (around CPU load at 95%)
- I would also closely look how the system behaves on the height load (request drop, higher response time, inconsistent behavior, ...)
- I would also look how the system behaves on 100% and higher load. Does it crash? Is there some auto-scaling implemented? How it behaves? Etc. but this is highly depended on how the system is designed.
- Test API standard response time
- This can and should be automated but I have no information about what response time is acceptable.
Test API for good practices
- Test if 4xx and 5xx responses returns JSON response
- Test if HEAD and OPTIONS request methods are available
Using Java with TestNG test framework
Description: when the limit
argument in URL is empty, not-a-number, negative number, decimal number, bigger then max integer (2 147 483 647) or smaller then min integer (-2 147 483 648) the server exception is invoked and return code is 500 Server Error
Tested cases: "", "aa", "agf5", "-5", "-245", "2.45", "8.012", "2147483648", "21474836471", "-2147483649", "-21474836485"
Steps to reproduce:
- Perform GET request to this URL https://api.nasa.gov/planetary/sounds?q=apollo&api_key=DEMO_KEY&limit=aa
- Check the response
Expected result: default limit (10) is used or response is 400 Bad request
Actual result: response is 500 Server Error
Probable cause: there is no check if the hits
key is present in array
Possible fix: instead of res = [x['_source'] for x in es_res['hits']['hits']]
use res = [x.get('_source') for x in es_res.get('hits', {}).get('hits', {})
Description: when Server Error happens the response contains stack trace
Step to reproduce:
- Perform request which causes
500 Server Error
for example with emptylimit
argument https://api.nasa.gov/planetary/sounds?q=apollo&api_key=DEMO_KEY&limit= - Check the response
Expected result: response contains just some general information about server experiencing error
Actual result: response contains actual stack trace
Probable cause: the DEBUG mode is turn on on production server
Possible fix: turn off the DEBUG mode in production
Description: when trying to search for some results with q
parameter the API returns the same set of results for every q
value. In the documentation there is no description of how the search feature should work but this behavior doesn't seems right.
Step to reproduce:
- Call the API with different
q
params (for example "apollo", "voyager", "car", "fridge", "asasdads", etc.) - Compare the results
Expected result: responses should differ and each response should only contains the results with the q
param value in description or name.
Actual result: the responses are identical even when the q
param is completely nonsense
Probable cause: search feature is not implemented
Possible fix: implement search feature
Description: responses contains headers with sensitive information. For example:
Server: openresty
- if somebody would decide to attack the API it is really valuable information to know on what server it is running. The attacker can focus on finding vulnerabilities in this serverVia: http/1.1 api-umbrela (ApacheTrafficServer [cMsSf ])
- same as previous the attacker now know that the server is using api-umbrella with ApacheTrafficServer to limit requests for API key
Step to reproduce:
- Perform any request to the API
- Examine the headers
Expected result: the headers containing sensitive information are not present
Actual result: the headers Server
and Via
contains sensitive information about underlying servers
Probable cause: the default server headers are used without proper server configuration
Possible fix: remove this headers in the server settings
- Error responses (4xx and 5xx) should return JSON instead of HTML
- It is convenient when the API is consistent and developers can relay on it that they gets proper description when when something went wrong
- HTML is good for browser exploring so maybe enable
Accept
header
- Enable
Accept
header when header isAccept: application/json
return JSON (even when 4xx or 5xx errors occurs) when header isAccept: text/html
return formatted JSON (for better reading in browser) and HTML error pages. The default should beAccept: application/json
- Enable HEAD and OPTIONS requests