Amazon Q Developer CLI - Context Grinding

I previously described the importance of refining your Q CLI configuration files for context/profiles/prompts in order to increase repeatability of results. I ran into this very issue just this week while creating a new configuration for performing a comprehensive feature branch analysis for MR reviewers.

Initially, I used an interactive session with Q CLI to begin documenting context for reviewing a feature branch. I spent perhaps an hour tops doing this and Q was able to generate context that I could use. It worked, and it worked in multiple repos. However, the content of the report it generated varied considerably from session to session. What I needed was actually a more static format for the report (Markdown), something skeletal that looked the same across Q sessions and across branches. Despite using some constraining language in the context, that was not sufficient.

So, I returned to an interactive session with Q and re-iterated the importance of repeatability of its resultant report structure. My guidance was:

Review the context file for Q in git-branch-analysis.md and create a new version that ensures repeatable results in terms of the output file structure. It is imperative that every branch that is analyzed is documented in the exact same manner each invocation. There can be no differences in the output file you generate for multiple stacks. The content may change given the details of the code changes, but the same headings and sections need to be present in the report file. Document your work on the changes comprehensively in a separate README file.        

After this, Q created a new version of the git-branch-analysis context. The context file was just over 300 lines of Markdown vs. 100 lines in the first version. Yes, that's a lot more of my context window being used, but it was still a fraction of the max window size for Claude Sonnet 3.7 so I was not deterred.

The result? Around 95% of the runs with the new context created identical report structure (compared to around 45% realized in v1). Here's a sample of the new context's strongly imperative wording to help guarantee repeatability:

You will perform a comprehensive analysis of a Git branch to understand its changes, purpose, and impact.

## CRITICAL GIT OPERATIONS REQUIREMENTS

**YOU MUST NEVER** create analysis based on assumptions or branch name inference. You MUST perform actual Git operations to analyze the branch content.

### MANDATORY GIT WORKFLOW

1. **Repository Verification**
   - Verify you are in a Git repository using `git status`
   - If not in a repository, request the user to navigate to the correct directory

2. **Branch Discovery and Fetch**
   - Use `git fetch --all` to ensure all remote branches are available locally
   - Use `git branch -a` to list all available branches (local and remote)
   - Verify the target branch exists before attempting checkout

3. **Safe Branch Checkout**
   - Check current branch status with `git status`
   - Stash any uncommitted changes if necessary with `git stash`
   - Checkout the target branch using `git checkout [branch-name]`
   - If branch doesn't exist locally but exists remotely, use `git checkout -b [branch-name] origin/[branch-name]`

4. **Branch Analysis Operations**
   - Use `git log --oneline main..[branch-name]` or `git log --oneline master..[branch-name]` to get commit history
   - Use `git diff --name-status main..[branch-name]` to get file changes
   - Use `git show --stat [commit-hash]` for individual commit analysis
   - Use `git diff main..[branch-name]` to examine actual code changes

5. **Content Examination**
   - Read modified files using appropriate file reading tools
   - Examine configuration changes, code modifications, and documentation updates
   - Analyze the actual content, not assumptions based on file names        

I could have spent hours upon hours of trial & error updating the context myself. However, by having Q do that work for me, it was done and over with fairly quickly.

Side note: Markdown is excellent for context because it allows for emphasis and highlighting that promote repeatability

My team is now using the git-branch-analysis configuration with Q to help them understand both the changes in a MR as well as a bunch of other useful information such as:

• Risk analysis
• Blast radius of changes
• Adherence to best practices (local and industry)
• Potential side effects

This is especially helpful for reviewers who are not familiar with a given configuration and do not have the same tribal and/or hands-on knowledge of the MR author.

Inspired by this success, I even built a second Q configuration using the lessons learned, to create a MR with a proper and invariant description structure, while Q fills in the details automagically after its commit analysis. Q leveraged an opensource CLI for our SCM that runs locally on my laptop in order to perform the MR creation (since Q is not nor will not be directly integrated in our SCM, this piece was the missing link to enable the workflow).

So, yes, crafting LLM context can be a grind, but always look for opportunities to let Q assist with your context development.