Indeed, my code has never been more rich in unit tests now that coding assistants make them so easy to add. If the coding assistant subsequently does something weird that changes program behaviour it knows immediately.
They check to make sure that the code continues producing the expected results. For example if I had some code that added up numbers, I might have a unit test that calls that function with the parameters “2” and “2” and checks to make sure the result is “4”. If something changes that makes it start returning something else then a flag is immediately raised.
Are these tests written by AI? Have you reviewed the tests? Do they cover all code paths and all the possible inputs and outputs? Do the tests have requirements cited (traceability)?
Unit tests have been around for a long time. They tended to be under-utilized in the past because they’re such a maintenance headache - changes / improvements in the production code that used to be a simple 5 minute thing can easily balloon to an hour or more of headache in necessary adjustments to perfectly good unit tests for “the old way” that happened to be impacted by the change.
Unit tests are a good way of keeping under-experienced programmers from accidentally breaking old things as they work on new things. They are VERY appropriate tools for LLM agent assisted development.
Boeing’s executives had motivated reasoning to skimp on actual engineering and give the results the exec wanted over the objections of actual engineers.
Sound familiar?
Frankly, no. LLMs are so sychophantic, there’s little evidence they would have flagged a warning that what he executives wanted was dangerous.
And no, LLMs are not just some tool. Never in human history has a hammer given it’s users psychosis.
The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
Indeed, my code has never been more rich in unit tests now that coding assistants make them so easy to add
Me as well. A year ago there was significant trouble with those unit tests not adding much value but that has improved a lot over the past several model updates. The unit tests aren’t 100% as good as they could or even should be (neither are human generated unit tests that take 10x as long to add) - but in terms of time invested vs value derived, they’re excellent value for effort.
If the coding assistant subsequently does something weird that changes program behaviour it knows immediately.
You can add deterministic testing to AI output though?
Indeed, my code has never been more rich in unit tests now that coding assistants make them so easy to add. If the coding assistant subsequently does something weird that changes program behaviour it knows immediately.
What do the tests do?
They check to make sure that the code continues producing the expected results. For example if I had some code that added up numbers, I might have a unit test that calls that function with the parameters “2” and “2” and checks to make sure the result is “4”. If something changes that makes it start returning something else then a flag is immediately raised.
Are these tests written by AI? Have you reviewed the tests? Do they cover all code paths and all the possible inputs and outputs? Do the tests have requirements cited (traceability)?
Unit tests have been around for a long time. They tended to be under-utilized in the past because they’re such a maintenance headache - changes / improvements in the production code that used to be a simple 5 minute thing can easily balloon to an hour or more of headache in necessary adjustments to perfectly good unit tests for “the old way” that happened to be impacted by the change.
Unit tests are a good way of keeping under-experienced programmers from accidentally breaking old things as they work on new things. They are VERY appropriate tools for LLM agent assisted development.
Absolutely not. In a world where hallucinating machines are writing code, humans should be writing the tests at a bare minimum.
AI writing code to pass tests written by AI will get people killed.
Boeing managed to kill plenty of people without a single bit of help from AI / LLMs in their MAX 10 software.
Used properly, LLMs could have articulated why their approach would lead to multiple crashes.
Used improperly, any tool is dangerous, from a chainsaw to a plastic spoon, the potential for injuries and deaths are unlimited.
Boeing’s executives had motivated reasoning to skimp on actual engineering and give the results the exec wanted over the objections of actual engineers.
Sound familiar?
Frankly, no. LLMs are so sychophantic, there’s little evidence they would have flagged a warning that what he executives wanted was dangerous.
And no, LLMs are not just some tool. Never in human history has a hammer given it’s users psychosis.
This is a completely nonsense argument.
The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
Me as well. A year ago there was significant trouble with those unit tests not adding much value but that has improved a lot over the past several model updates. The unit tests aren’t 100% as good as they could or even should be (neither are human generated unit tests that take 10x as long to add) - but in terms of time invested vs value derived, they’re excellent value for effort.
About 90% of the time for me now, in Opus 5.5.