#ASTTokens

danzindanzin
2023-09-10

With support from , we can then use to collect Raw (LOC, comments, blank lines, etc.) metrics from functions and methods.

After adding support for to annotate source code with these metrics, we can display them inline with help from HTML and JS, as in the attached images.

Each line displays the selected metric (or all metrics) on the left. It allows quickly finding, e.g., code with too few comments or blank lines versus lines of code.

5/5

A view of highlighted Python source code shows parts of three methods. On the left, a column of blue numbers represent line numbers, while black numbers represent the Lines of Code of each method.

On the top there are buttons labelled "Toggle", "LOC", "LLOC", "SLOC", "Comments", "Multi", "Blank", "Single", "All". The button with "LOC" in it is selected.A view of highlighted Python source code shows parts of three methods. On the left, a column of blue numbers represent line numbers, while black numbers represent the number of comments of each method.

On the top there are buttons labelled "Toggle", "LOC", "LLOC", "SLOC", "Comments", "Multi", "Blank", "Single", "All". The button with "Comments" in it is selected.A view of highlighted Python source code shows parts of three methods. On the left, a column of blue numbers represent line numbers, while seven columns of black numbers represent different metrics of each method, corresponding to the buttons above.

On the top there are buttons labelled "Toggle", "LOC", "LLOC", "SLOC", "Comments", "Multi", "Blank", "Single", "All". The button with "Comments" in it is selected (but all metrics are shown).
danzindanzin
2023-09-08

We can now include trailing comments in functions, but want to avoid including extraneous comments in the roundtripped code. To achieve that, we need to stop adding comments as soon as we see one that is less indented than the first line of the function we're interested in.

So we:
1- Recover the code from an node using .get_text().
2- Add any trailing comments after the last token of the function.
3- Unless the comment is less indented: we stop adding in that case.

4/n

danzindanzin
2023-09-08

Since allows getting the next token from end of the function node, it's possible to fetch the next tokens if they are comments, and stop at the last comment before a token of another type.

Using the last comment token's end position, we fetch the source code between what ASTTokens.get_text() gave us and that last comment. That allows a full roundtrip of code that includes trailing comments.

However, that will also include any comments after the function body 😞

3/n

danzindanzin
2023-09-08

So, for to calculate LOC and number of comments for each function, it needs a way to roundtrip code with all comments included.

Enter , which "annotates Python abstract syntax trees (ASTs) with the positions of tokens and text in the source code that generated them".

However, trailing comments get lost when using ASTTokens.get_text(), because the comment token isn't linked to the function node. So no roundtrip yet.

2/n

github.com/gristlabs/asttokens

Client Info

Server: https://mastodon.social
Version: 2025.04
Repository: https://github.com/cyevgeniy/lmst