Phoenix at RubyConf

We wrapped up another fantastic RubyConf here in Chicago! So much learning and fun packed into three days. Most importantly, we met new friends and tried Phoenix out on a whole bunch of Rails apps. Read on to learn the progress we've made and about what's new in the tech. 

Our mission is to generate Rails tests that are as good or better than the ones you'd write yourself. These updates are intended for people who want to see a tool like this exist in the world.  

Technical Updates

Update: We've got the first version of RailsGrokker running! RailsGrokker builds a dependency graph of a codebase to better determine the right context for the LLM. Many of the people who worked with us at RubyConf weren’t able to share their entire codebase, so they AirDropped us a few models. Since we didn’t have a full codebase, we weren’t able to use RailsGrokker, but we stuck those models into a fresh Rails project and Phoenix was still able to generate tests. Then we sent the tests back to them so they could run them in the context of their own projects.  It was great fun working with people to review the tests and fix the issues.

Challenge: Even with RailsGrokker providing better context, the generated tests often failed in similar ways. The biggest challenge has been when a test needs to create a complex object with multiple dependencies. Phoenix still gets this wrong much of the time. We found that including factories and existing tests in the context helped improve the outcome. Over the next couple of weeks, we’ll focus on overcoming these issues by incorporating the feedback from our RubyConf friends into Phoenix and RailsGrokker.

How You Can Help: We’re still interested in learning more about other people doing similar work. At RubyConf, we learned about Sublayer, a ruby AI agent framework that they’ve used for test generation. A friend told us about StartEarly, a test generation tool focused on JavaScript testing. What else have you come across recently?

Business Updates

Update: Here at RubyConf we made a simple proposition: share some Rails code with us and we'll give generate some tests for you. That was enough to keep us busy for an entire Hack Day here on Day 2. We learned so much and had a lot of fun putting Phoenix to the test. Thanks to Bekki, Brenda, Chris, Esteban, Bill, and Peter!

Challenge: We are still looking to secure design partners for Phoenix. We’re reaching out to Rails developers who believe in Phoenix’s potential and want to bring the benefits of streamlined, AI-assisted testing to their business.

How you can help: If you or someone you know is working on a Rails code base in need of automated tests, let us know! We are actively looking for partners to help us improve Phoenix. You get free tests and we get invaluable feedback - everybody wins!