BetterHook - A framework for systemic code evaluation and surfacing of regression-type code-level problems

Hey there.

I assume you are familiar with the idea of web hooks (like, GitHub post-commit hooks?).

Typically, post-commit hooks that we setup as a part of testing or CI/CD pipelines are working to ensure any events (current and future) are reacted to if they are subscribed to.

Not all the organizations (a lot of times, the fast moving ones) have a mechanism to narrow down when a problem became a part of the code-base even when they know there is a problem in a current code-base. This opens up the scope of similar such events at different areas in the organization, especially the ones which are not well automated or even monitored.

I wonder if we have a service (I am not aware of one) which works like this, once its setup is in place:

• Setup a bunch (whatever we like, say: static analysis, unit tests, regression tests, et cetera) of post-commit hooks for any given organization/repository. This action has taken place at a given point in the timeline i.e. this point is not the start of the timeline!
• A typical action takes place (post-commit hook) via event subscription
• A much deeper task is initiated in the background which handles all the previous commits (or some selected bunch via some characteristic) as if the post-commit hook existed for all of them. This obviously needs some contextually necessary optimization to make sure the efforts won't take all the time in the universe, give or take, for completion
• This deeper task (say, D1R1 s.t. D refers to deeper task and R refers to repository one), with false positive reduction feedback aka fprf in mind, surfaces appropriate potential issues of whatever kind the tests where supposedly created for.

One might find this as a ridiculous way of slowing down the development pipeline but the learning we are going to get out of this is going to be a great feedback tool for all the developers in the company. Some other benefits, I see, can be like this:

• Imbue better development habits.
• Get a picture of how the overall security posture has moved in the products.
• Verify API contracts.
• Improve test cases and an easier way to amend them with a larger impact.
• Save a lot of money which would be otherwise invested in tools doing some of the things from this list.