Generative AI in ROS 2 Codebases? Oh My!

[bacon4life]

That is quite a strict, or hard to follow policy. May I apologize for the nitpicking.

While I agree on the overview:

@gbiggs

In general, generative AI is a benefit to making better PRs … two potential liabilities:

1. Legality …
2. The generated content may not do what is expected …

---

Feedback on suggested rules on source code PRs

@gbiggs

1. The PR title and description must be entirely hand-written. … submitter may not understand their own PR.

As I said in my previous post, I don’t really like the hard “must”. I’d agree if it had an “In general” beforehand.

@gbiggs

2. The PR description is required to list the provenance of all code in the PR.

Nothing to disagree on!

@gbiggs

3. For a PR that includes source code from generative AI, the PR description must list approximately what proportion of the changed lines of the PR contain generated code, what generative AI models/tools were used to produce them, and for what purpose. For example:

   1. if CLion’s AI-based autocomplete is used, that should be listed as “CLion autocomplete - completing function parameters - all function calls;
   2. if ChatGPT was used to debug some hand-written code, that should be listed as “ChatGPT - debugging of hand-written code - two lines of algorithm”; and
   3. if ChatGPT was used to generate an entire algorithm implementation, that should be listed as “ChatGPT - generated algorithm implementation - 75% of lines”.

I think this is a bit too much. Ultimately the goal is to reduce the 2 risks posed in the overview, and I struggle to understand why this level of detail is needed.

@gbiggs

4. For each git commit in the PR, if the content of that commit contains AI-generated source code, the commit message must note this.

Really? Some people commit very often, especially in development stage. This will really disturb their workflow. And if the answer is to squash the commit, then what’s the difference between squashing and only writing in the PR?

@gbiggs

5. All source code derived from generative AI must be fully tested with 100% line coverage as a minimum, and ideally full MC/DC coverage. Tests will ideally be hand-written, or at least hand-designed.

Again, I feel “All” is too strong. Sorry for nitpicking, but what about some obvious AI-assisted autocompletes? What’s the difference between it and an experienced programmer typing from habit?
Also, what’s the definition on “100% line coverage”? Does manually testing count? I’ve never had good experiences with projects chasing line coverage, especially 100%.

@gbiggs

6. If a maintainer suspects that a PR may have been substantially generated by an AI-based source code generating tool, or suspects that the submitter has not been honest or accurate in the PR description about what was generated by AI, for what purpose, and what proportion of the PR, the maintainer has the right to request a walkthrough of the PR’s code by the submitter using a teleconference tool.

No objections on that.

---

Feedback on suggested rules on documentation PRs

@gbiggs

1. The PR title and description must be entirely hand-written. Using generative AI to produce the title and/or description indicates that the submitter may not understand their own PR.

Same as the source code ver., I feel “must” too strong. Especially for a documentation PR, I can see some being just as below, which I think is 100% fine to be auto-generated.

Add documentation for class/functions in file x

List of class/functions with documentation added:
...

@gbiggs

2. The PR description is required to list the provenance of all content of the PR.

No objections!

@gbiggs

3. For a PR that includes documentation from generative AI, the PR description must list approximately what proportion of the changed lines in the PR contain generated documentation, what generative AI models/tools were used to produce them, and for what purpose.
4. For each git commit in the PR, if the content of that commit contains AI-generated source code, the commit message must note this.

Same problems with the source code version.

@gbiggs

If a maintainer suspects that a PR may have been substantially generated by an AI-based documentation generating tool, or suspects that the submitter has not been honest or accurate in the PR description about what was generated by AI, for what purpose, and what proportion of the PR, the maintainer has the right to request a walkthrough of the PR’s documentation changes by the submitter using a teleconference tool.

Same as the source code version, no objections.

---

Just for one last note, if we value the maintainer’s time and sanity, the last thing we want is suspecting if the PR writer is lying (maybe due to too much procedures), or (god forbid!) a video call.
Thus I think making life easier for the people writing PRs is equally important as to make life easier for the maintainers. (Although I might be too biased )

---

@peci1

I’d just add a requirement that for a docs PR, the submitter has manually read all the generated docs and verified their accuracy. This is important because otherwise the docs won’t fit the code, which is worse than no docs.

This is a very good point!
Now that I think of it, there’s also the problem with non-native (or fluent) English speakers.
More often than not, their only choice is a translation service - which isn’t really helpful.
That’ll be a discussion for another thread, but I think we should keep in mind that not everyone is good at English. (Although like it or not, there’s always the option to ignore those people)