How to intercept, analyze and log secure HTTP requests using .Net core Web API
I was facing a problem where a client app was making an HTTPS request to a server. The server was returning a 400 error.
Usually, these errors are associated with a bad request because the client is sending bad heathers or the body is malformed.
The problem was that the owners of the client app were saying that the application was correct and there was an error in the server.
The server owners were saying that the server was correct and the problem was in the client app.
When making the request using Postman everything was working fine and even making a request inside the RedHat server where the app client lived using CURL the request was a success. With this, we were discarding that some middleware systems like the firewall were adding heathers to the request that were affecting it.
So we were almost sure that the error was in the client app but nothing appeared in the client app logs neither the Server app logs.
Another problem that we had was that the client app can only make HTTPS request making unable to capture the traffic and inspect it using a sniffer like Wireshark.
The way I solve the problem of capturing the request to analyzing it was using Ngrok and a .Net Core 3.1 Web API.
First I use Ngrok that is an application that opens an SSH tunnel between your computer and their servers in the cloud. It opens communication in their servers and redirects the traffic through the tunnel to my Web Application.
This Web API that I created using .Net Core that exposes a service that caches any request and prints the information in a log file.
1.- Install Ngrok on your PC
2.- Run the Ngrok app in your local
ngrok http https://localhost:5001
- Ngrok is the command
- HTTP is the protocol that is going to be sent through the tunnel
- https://localhost:5001 is where the traffic is going to be redirected.
3.- Create the web API that is going to expose the service.
Create the Web API with the next structure
Add the NLog.Extensions.Logging using Nugget Package
Add the middleware and the ILog interface in the startup class
The important part is the middleware that is the one that is going to intercept any request a log it
using Microsoft.AspNetCore.Http;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using WebApplicationRick.Log;
namespace WebApplicationRick
{
public class RequestResponseLogginMiddleware
{
private readonly RequestDelegate _next;
private readonly ILog _logger;
public RequestResponseLogginMiddleware(RequestDelegate next, ILog logger)
{
_next = next;
_logger = logger;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
try
{
await _next(context);
}
finally
{
_logger.Information(
string.Format("Request {{URL}} -> {0} {1} {2} {3}",
context.Request?.Method,
context.Request?.Host,
context.Request?.Path.Value,
context.Request?.QueryString.Value));
_logger.Information(
string.Format("Request {{More info}} {{Protocol}} -> {0}, {{ContentType}} -> {1} ",
context.Request?.Protocol,
context.Request?.ContentType));
foreach (var heather in context.Request.Headers)
{
_logger.Debug("HEATHER -> " + heather.Key + " " + heather.Value);
}
using var reader = new StreamReader(context.Request.Body, Encoding.UTF8);
var reqBody = await reader.ReadToEndAsync();
_logger.Information($"{{RequestBody}} -> {reqBody}");
}
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
request.EnableBuffering();
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
await request.Body.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
var bodyAsText = Encoding.UTF8.GetString(buffer);
request.Body.Position = 0;
return $"{Environment.NewLine}{Environment.NewLine}{Environment.NewLine}{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}{Environment.NewLine}{Environment.NewLine}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
}
Once it is ready to run the application, by default it will start licensing in a secure port https://localhost:5001
4.- Call the service through the Ngrok service by using the URL that was provided by Ngrok local application.
Ngrok will use a valid Certificate so you will not have any problem with the certificate. As a note, Ngrok uses wildcard certificates that can be invalid in some applications.
https://d37784bb8e3f.ngrok.io/oauth2/api/v1/token?grant_type=client_credentials
We can see the response in the console and the web interface.
5.- The logs will be sent to a file where I am logging the request info: method, URL, status code, and all the Heathers.
