{"error": "forbidden"} datadog postmortem

Issue Summary

From 6:00 pm to 7:00 pm GMT-5, after creating monitoring to the server (Create an account, install datadog-agent in the server), the creation was correct, but at the moment to validate the information using the API of datadog:

curl -X GET "https://api.datadoghq.com/api/v1/hosts" 
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
        

change by correct API key and Application key, we get how to answer:

{"errors": "forbidden"}        

impacting the monitoring of server, were not possible to see the information of the server in the datadog. affecting the data analytics; the root cause was subdomain.

Timeline

• 5:40 PM - Initialization of configuration and creation of datadog agent.
• 6:00 PM - Finish creation and configuration of datadog agent (account and installation).
• 6:05 PM - Execution of curl command to get information (Failed)
• 6:08 PM - Review of steps do it (seem the steps was correct)
• 6:15 PM - Execution of curl command to get information (Failed)
• 6:18 PM - Comparation with other creations of datadog. (how was the issue detected)
• 6:30 PM - Differences found in the subdomain. (when was the issue detected)
• 6:40 PM - Initialization of configuration and creation of datadog agent with new subdomain
• 6:55 PM - Finish creation and configuration of datadog agent with new adjustments (account and installation).
• 7:00 PM - Execution of curl command to get information (Success)

Root cause and resolution must contain

We see that the problem is in the moment of install the datadog-agent in the server, we use next command to install datadog agent:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=... DD_SITE="us5.datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"        

We see in the DD_SITE variable, it had the subdomain US5 ("us5.datadoghq.com"), and the URL of the site where the account was created, had the same subdomain:

https://us5.datadoghq.com/signup/agent#ubuntu [EXAMPLE]
https://us3.datadoghq.com/signup/agent#ubuntu [EXAMPLE]        

When comparing with another datadog installation, we see that in the correct installation the DD_SITE variable had no subdomain (us3, us5, etc) but only "datadoghq.com", and in the site URL this app as a subdomain:

https://app.datadoghq.com/signup/agent#ubuntu [EXAMPLE]        

The problem was solved by adjusting the URL at the time of creating an account on the website by putting the app subdomain in the URL:

change https://www.datadoghq.com/ to https://app.datadoghq.com/        

With this, the subdomain setting will be none by default, and when installing the datadog agent on the server, the command on the web page will be:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=... DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"        

Without subdomain, and the URL of web page will:

https://app.datadoghq.com/signup/agent#ubuntu        

and by executing the following command, we will have a correct answer:

curl -X GET "https://api.datadoghq.com/api/v1/hosts"
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}"         

Corrective and preventative measures must contain

You don't need this, with the previous steps, you can fix the problem with datadog, or maybe not, because the software is perfect, but the developers not :)