Use coverage from previous commit(s)

Hello!

We use semantic-release to publish new releases of our software, but the way Codacy handles coverage does not play nice with it.

After each commit that should trigger a release (fix, feat, etc.), a workflow runs, creating a new commit, which increments version numbers and updates CHANGELOG.md, and then publishes a release on GitHub. The commit created in this process has [skip ci] in the message, so this commit will not trigger any GitHub Actions builds, and thus not have a coverage report associated with it.

Here’s an example where you can see what our repositories’ commits look like:

I’ve already added a workaround to this repository for coverage disappearing after committing something that should not trigger tests by caching coverage files, skipping tests and then uploading the old coverage again. This does not work for the [skip ci] commits, though, as the workflow does not run at all. I really don’t want to remove the [skip ci] tag from those commits, because this introduces another layer of workarounds to get this working, and the preferred behaviour is to NOT run workflows at all when they’re not necessary. Other coverage CI tools I’ve used do not have this problem.

We’d like to be able to use one single service for both static analysis and coverage, but until this works properly we can’t have the “Codacy Coverage Report” check block pull requests, and we’re still forced to use another service to manage coverage if we want to keep using Codacy.

PR example: feat: add shipments by EdieLemoine · Pull Request #24 · myparcelnl/pdk · GitHub

At the time of writing, we have a commit in main that has been analysed, so the Codacy Coverage Report has passed.

As soon as a new release commit is added to main and the PR is rebased, this check will be stuck in “pending” forever, and the Codacy dashboard will say no coverage has been uploaded, while the coverage rate in the “Quality evolution” block does still say 100%.

I really hope this can be resolved.

Thanks in advance!

Edie

Developer at MyParcel

3 Likes

Hey @EdieLemoine :wave: I’m the Product Manager of our Coverage product.
Thank you very much for your feedback.

Do you mind if I ask a few questions to understand your situation a bit better?

Question 1: Is it essential for you to have the Codacy’s checks mandatory on GitHub?
Question 2: You mentioned other coverage CI tools. What do they do that fits your use case better?

Hi @fabio,

1: Yes, absolutely. We want to have the strictest possible pull request flow.
2: I’ve tried a few others in the past but have stuck with Codecov.

  • Their pull request checks do work
  • They post a clear comment on the pull request explaining the coverage report and/or any issues uploading/reading coverage.
  • I can upload coverage from GitHub actions without providing credentials, which also avoids issues with dependabot (for which I’m aware there’s a workaround) and external contributors.

I’m still having the problem I posted about, but now even the Static Code Analysis check is pending forever. I’m also not seeing new commits, branches or any other updates on the Codacy dashboard, while your status page shows no issues.

We now have codacy/codacy-analysis-cli-action running, I’ve tried with and without the sarif upload and GitHub code scanning, and with “Run analysis on your build server” turned on and off. I just want this to work, maybe you or someone can enlighten me about what I’m doing wrong?

Example PR with pending checks:

Dashboard doesn’t show a new commit I made about 20 minutes ago, and above PR is also not showing up.

Hi @EdieLemoine !

I just took a look at your repo and your PR’s seem to be appearing right now. Around the time you reported this we were facing a spike in usage that can cause some delays on a repo’s update.

Can you please confirm if you still have your status waiting for Codacy?

Hi @Ivo, it did eventually resolve itself on the PRs that were open, but now I’m experiencing it again on another one: feat: add fulfilment order repository by Mark-Ernst · Pull Request #33 · myparcelnl/pdk · GitHub

It says “Analysis failed” on the Codacy dashboard, but when I click it it says it’s analyzed, and the logs show all steps were successful. By the time you see this it may resolve itself as I’m pushing to the main branch again, which will rebase that PR. It does happen a lot, even unrelated to what I wrote in this post. I really just want this all to work smoothly.

Thanks for responding!