scala-steward-org / scala-steward Goto Github PK
View Code? Open in Web Editor NEW:robot: A bot that helps you keep your projects up-to-date
License: Apache License 2.0
:robot: A bot that helps you keep your projects up-to-date
License: Apache License 2.0
Currently scala-steward checks every repository for new dependencies on every run. Checking for updates is a costly operation since it means cloning the repository, starting sbt, and checking for library and plugin updates. Even for small repositories which are up-to-date this process takes roughly a minute on my server. This is not viable if scala-steward should be able to handle 1000+ repositories which I very much like to achieve.
My current idea to reduce the overall workload is the following:
I propose to skip checking repositories where scala-steward would not open new PRs or update existing PRs. This would require to persist some infos about the repos scala-steward has worked on. E.g if we know of each repo
we should be able to decide if we need to process a repository without cloning it and running sbt dependencyUpdates
. The SHA1 can be used check if the upstream repository has changed and our existing PRs need to be updated or if the list of dependencies has changed.
Before working on any repository we would need to check if any of the libraries or plugins scala-steward knows about has a newer version and then only process repositories whose dependencies have updates.
Could we do that?
To identify the batch we could label all pull requests made by the bot by the current build number.
If all were successful and merged we could trigger somehow a release?
Challenges
- addSbtPlugin("com.geirsson" % "sbt-ci-release" % "1.2.1")
+ addSbtPlugin("com.geirsson" % "sbt-ci-release" % "1.2.5")
Not sure why sbt-ci-release change was made here.
sbt version update should be done using project/build.properties
.
See https://github.com/pdalpra/gh4s/pull/15 for an example.
... instead of adding it to repos.md
. I guess this requires making some kind of GitHub app for Scala steward. The only thing I'd need from GitHub would be a list of repos that have Scala steward "installed".
Are there facilities in scala-steward to configure version ranges?
For example lightbend/ssl-config#133 updates joda-time from 2.9.9 to 2.10. Now, leaving aside that right this moment I don't know if that's a safe update for ssl-config, assuming we can't do that upgrade how do I inform scala-steward that for the moment we want only updates in the 2.9.x version series?
(Thanks for the bot, looks like a great addition to the ecosystem!)
Hi @fthomas
Thanks for this great bot. I love it.
I have some interest in making scala-steward
work for repositories that don't have the luxury of allowing forks
and limits users to work with branches out of one main repository.
I tried to tweak scala-steward to my use-case and got it to work. (Needs some clean up)
ArulselvanMadhavan@18a3b11
If you are interested in adding this as a feature behind a config flag(or some other better option), let me know. I would like to contribute towards that effort.
Hi @fthomas, I've discovered that firejail doesn't work in Docker:
$ firejail echo hello
Warning: an existing sandbox was detected. echo will run without any additional sandboxing features
Running it with the --force
flag, which is supposed to disable the PID namespace checking, results in the following error:
$ firejail --force echo hello
Error clone: main.c:2519 main: Operation not permitted
Additionally, the --force
flag was removed in firejail 0.9.54 and apparently it is not possible to run firejail in Docker anymore (I asked about this recently netblue30/firejail#2579).
I think we should remove firejail from scala-steward docker image and force it to run without the sandbox when the command isn't found or make it explicit in the docs that the sandboxing features don't work when running scala-steward in Docker.
We could group updates with the same groupId
and the same currentVersion
and newerVersions
as updates that need to be applied in lockstep.
Is there a way to integrate scala-steward with scalafix, so that when a library maintainer supplied a scalafix migration from one version to another (such as the one in the works for fs2 0.10 to 1.0), that steward can automatically apply it in the PR along with the build change?
We have a custom sbt plugin that is hosted at the companies artifactory. I think the resolver that is defined in plugins.sbt is not being used when checking updates for this plugin.
Our plugins.sbt looks like this:
externalResolvers := Seq("Company Repo" at "https://repo.company.io/artifactory/")
addSbtPlugin("io.company" % "company-plugin" % "0.1.15")
The error we’re getting when running Scala Steward
[info] Loading global plugins from /root/.sbt/1.0/plugins
[info] Loading settings for project updates-build from plugins.sbt ...
[info] Loading project definition from /home/jenkins/workspace/updates/project
[info] Updating ProjectRef(uri("file:/home/jenkins/workspace/updates/project/"), "updates-build")...
[warn] module not found: io.company#company-plugin;0.1.15
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== local: tried
[warn] /root/.ivy2/local/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/io/company/company-plugin_2.12_1.0/0.1.15/company-plugin-0.1.15.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /root/.sbt/preloaded/io.company/company-plugin/0.1.15/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////root/.sbt/preloaded/io/company/company-plugin_2.12_1.0/0.1.15/company-plugin-0.1.15.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: io.company#company-plugin;0.1.15: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
And thanks for the awesome project. We're running it successful in production for a few weeks on 10+ repos!
Sometimes scala-steward updates version numbers of dependencies that are similar to the current update but not part of it. They are similar because these dependencies use the same version and their artifactIds have the same prefix. One example is http4s/http4s#2343.
We could fix such errors by trying to update version numbers in two steps. The first step would try to update only lines that contain the exact groupId, artifactId, and version. If this step changes any lines we're done. If it doesn't yield any changes we would use the current strategy for updating version numbers. In the case of http4s/http4s#2343 this strategy would have updated the version numbers of jawn-json4s and jawn-play but not of jawn-fs2 because it has a different groupId.
ProcessAlg.execSandboxed
is an alternative to ProcessAlg.exec
that runs programs in a sandbox. If one fully trusts the repos scala-steward is working on execSandboxed
could be just an alias for ProcessAlg.exec
. It would be great to have a config option for this case.
FilterAlg
currently discards updates where nextVersions.head
is a known bad version. Here are some examples. Instead of discarding them, we could also just remove the bad version in nextVersions
and update to the new nextVersions.head
. If nextVersions
only contains one element, we would again discard that update.
We'd like to leverage this awesome project to update the dependencies on our internal project dependencies. To do that, scala-steward must support GitLab. I've started to work on a native implementation.
I'm opening this MR just to leave a trace of that on the internet. That way, if someone else planned/was planning or is already doing it, we can coordinate better.
Thanks again for the project, it is really a breeze to work it.
see https://github.com/maximn/TaxCalculator/pull/1/files
The diff is:
- libraryDependencies ++= Seq("org.specs2" %% "specs2-core" % "3.+" % "test")
+ libraryDependencies ++= Seq("org.specs2" %% "specs2-core" % "4.3.4
Abstract:
When a ModuleID
is written with version detached into another variable, scala-steward
seems to be unable update the version:
val cats_effect_version = "1.1.0"
val cats_effect = "org.typelevel" %% "cats-effect" % cats_effect_version
After importing the Izumi Project here, we've got two PRs from scala-steward: 7mind/izumi#460 7mind/izumi#461 both only concern ModuleID's that were written all at once while the large IzumiDeps
object above was not processed. (Note, there's another bad case in IzumiDeps
- constructing ModuleIDs with .map
- however, this case likely can't be handled sanely, so we'll just flattenize those.)
It looks like scala-steward doesn't try to update the scala version.
For example in the build.sbt of metrics-scala we have: scalaVersion := "2.12.6"
and crossScalaVersions := Seq("2.11.12", "2.12.6")
. Version 2.12.7 is already out.
This has gone terribly wrong: lightbend/paradox#263
I have a project, which only has the SBT project layout at branch sbt
, see here: https://github.com/ReactivePlatform/Pragmatic-Scala
, seems like steward needs an update to support this kind of usage.
So that we don't hit the rate limit, see https://developer.github.com/v3/#rate-limiting
I didn't fully understand how caseapp
handles Boolean
when I named the config as execSandbox
and set the default to true
. This issue is to rename the config name from execSandbox
to disableSandbox
.
I'd like to use this on my company's codebase to help save maintenance time. Is that possible?
It would be nice for scala-steward to be able to work with mill and maybe other build tools as well. As I understand this requires three components:
I believe the 3) could be shared between build tools, but I dont know hard would that be.
It would be nice to learn how hard 1) and 2) could be.
I assume 1) is as simple as executing proper mill command and parsing the results.
2) sounds like could be done with scalafix, but I dont know it well enough to say something for sure.
Scala steward should also propose sbt updates and not only library and plugin upates.
If a repository has more than n
(say 30) open pull requests by Scala steward it makes sense to ignore this repository until the open PR count drops below the threshold n
. Many open pull requests are a sign that a repository is abandoned and working on that is just a waste of resources.
... in a perfect world to its release notes.
If a project uses multiple versions of the same dependency, Scala Steward does a bad job at handling them in a coordinated fashion which results in bad PRs like circe/circe-jackson#36 or scala/scala-collection-compat#432. This is a common situation for projects that support multiple Scala versions or when a project provides integration for multiple versions of another library (e.g. circe-jackson25, circe-jackson26, etc.).
As suggested by @fwbrasil in zio/zio-quill#1195 (comment).
scala-steward is capable of running in a project that uses maven for project definition by using https://github.com/sbt/sbt-pom-reader to start sbt and get dependencies. If this can't be done reliably, it may need to be opt-in on a project-by-project basis.
Once the dependencies are enumerated, applying a different set of heuristics to the sbt version replacements seems straightforward enough.
https://developer.github.com/enterprise/2.15/v3/enterprise-admin/#endpoint-urls
REST API endpoints—except Management Console API endpoints—are prefixed with the following URL:
http(s)://hostname/api/v3/
This PR WellFactored/clovis#15 updates the version number of enumeratum-circe
by changing the value of enumeratumVersion
from 1.5.13 to 1.5.19. However, that val is also used to define the version of the core enumeratum
library, which does not have a 1.5.19 version, so the subsequent build breaks.
... instead of regular expressions.
Like the Play framework, which put the real sbt project in /framework
folder.
It tried to "update" from 3.0.0-RC2-d0feeba
to 3.0.0-fbcb270
, which is actually older.
To be honest I find this versioning scheme confusing myself, so I'm not surprised it got tripped up by it 😉
Since #197 is merged, scala-steward
can work on any repo regardless of the git workflow an organization chooses to use. So, companies now have the option of using it on their internal private repos. This issue is to see if there is any interest in adding a list of companies that are using scala-steward
somewhere in the repo to be well informed of the community that is benefiting from scala-steward
.
Something similar to https://github.com/circe/circe#community
@fthomas Thoughts?
GUI git clients that show gravatars for commits would look even better with a Gravatar for scala-steward :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.