Use coverage from previous commit(s)

Hello!

We use semantic-release to publish new releases of our software, but the way Codacy handles coverage does not play nice with it.

After each commit that should trigger a release (fix, feat, etc.), a workflow runs, creating a new commit, which increments version numbers and updates CHANGELOG.md, and then publishes a release on GitHub. The commit created in this process has [skip ci] in the message, so this commit will not trigger any GitHub Actions builds, and thus not have a coverage report associated with it.

Hereā€™s an example where you can see what our repositoriesā€™ commits look like:

Iā€™ve already added a workaround to this repository for coverage disappearing after committing something that should not trigger tests by caching coverage files, skipping tests and then uploading the old coverage again. This does not work for the [skip ci] commits, though, as the workflow does not run at all. I really donā€™t want to remove the [skip ci] tag from those commits, because this introduces another layer of workarounds to get this working, and the preferred behaviour is to NOT run workflows at all when theyā€™re not necessary. Other coverage CI tools Iā€™ve used do not have this problem.

Weā€™d like to be able to use one single service for both static analysis and coverage, but until this works properly we canā€™t have the ā€œCodacy Coverage Reportā€ check block pull requests, and weā€™re still forced to use another service to manage coverage if we want to keep using Codacy.

PR example: feat: add shipments by EdieLemoine Ā· Pull Request #24 Ā· myparcelnl/pdk Ā· GitHub

At the time of writing, we have a commit in main that has been analysed, so the Codacy Coverage Report has passed.

As soon as a new release commit is added to main and the PR is rebased, this check will be stuck in ā€œpendingā€ forever, and the Codacy dashboard will say no coverage has been uploaded, while the coverage rate in the ā€œQuality evolutionā€ block does still say 100%.

I really hope this can be resolved.

Thanks in advance!

Edie

Developer at MyParcel

3 Likes

Hey @EdieLemoine :wave: Iā€™m the Product Manager of our Coverage product.
Thank you very much for your feedback.

Do you mind if I ask a few questions to understand your situation a bit better?

Question 1: Is it essential for you to have the Codacyā€™s checks mandatory on GitHub?
Question 2: You mentioned other coverage CI tools. What do they do that fits your use case better?

Hi @fabio,

1: Yes, absolutely. We want to have the strictest possible pull request flow.
2: Iā€™ve tried a few others in the past but have stuck with Codecov.

  • Their pull request checks do work
  • They post a clear comment on the pull request explaining the coverage report and/or any issues uploading/reading coverage.
  • I can upload coverage from GitHub actions without providing credentials, which also avoids issues with dependabot (for which Iā€™m aware thereā€™s a workaround) and external contributors.

Iā€™m still having the problem I posted about, but now even the Static Code Analysis check is pending forever. Iā€™m also not seeing new commits, branches or any other updates on the Codacy dashboard, while your status page shows no issues.

We now have codacy/codacy-analysis-cli-action running, Iā€™ve tried with and without the sarif upload and GitHub code scanning, and with ā€œRun analysis on your build serverā€ turned on and off. I just want this to work, maybe you or someone can enlighten me about what Iā€™m doing wrong?

Example PR with pending checks:

Dashboard doesnā€™t show a new commit I made about 20 minutes ago, and above PR is also not showing up.

Hi @EdieLemoine !

I just took a look at your repo and your PRā€™s seem to be appearing right now. Around the time you reported this we were facing a spike in usage that can cause some delays on a repoā€™s update.

Can you please confirm if you still have your status waiting for Codacy?

Hi @Ivo, it did eventually resolve itself on the PRs that were open, but now Iā€™m experiencing it again on another one: feat: add fulfilment order repository by Mark-Ernst Ā· Pull Request #33 Ā· myparcelnl/pdk Ā· GitHub

It says ā€œAnalysis failedā€ on the Codacy dashboard, but when I click it it says itā€™s analyzed, and the logs show all steps were successful. By the time you see this it may resolve itself as Iā€™m pushing to the main branch again, which will rebase that PR. It does happen a lot, even unrelated to what I wrote in this post. I really just want this all to work smoothly.

Thanks for responding!