Possible to publish WIP to maven-central?

[daniel-sudz]

The Github Artifacts require authentication for read. This makes them hard to use seamlessly in other open source projects. Would it be possible to cross-publish to maven central or another maven-compatible repo with read enabled without auth?

[daniel-sudz]

@cwensel I would also be happy to help add github-actions worflows for CI on PR and CD for wip branches to make it easier to integrate

[cwensel]

this is a good request.

I could publish to maven snapshot repo, but unsure of the rules and accessibility (re requiring creds).

unfortunately the builds take many hours, and I haven't tried to convince github actions to perform the builds yet (not even sure I can get a durable build number for the wip name).

the other solution is to simply release more often. there are only a few things I would have liked to get into 4.1/4.5, but unsure I have time to tackling them and probably should defer them.

is this currently a blocker?

[daniel-sudz]

Not a blocker, but nice to have (I have a local workaround for now). I can comment on the github actions:

the modern timeout is quite generous (and could be expanded to forever it seems like): https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes and if sharded by module should finish in time.

jobs.<job_id>.timeout-minutes
The maximum number of minutes to let a job run before GitHub automatically cancels it. Default: 360

If the timeout exceeds the job execution time limit for the runner, the job will be canceled when the execution time limit is met instead. For more information about job execution time limits, see "[Usage limits and billing](https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits)" for GitHub-hosted runners and "[About self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners/#usage-limits)" for self-hosted runner usage limits.

I've implemented a github-actions on scalding their builds are also quite large and it seems to work fine. See twitter/scalding#1963

I can throw a draft for github actions if you would like

[daniel-sudz]

The way I understand is that github-actions runs in the context of a user-name. So if I fork cascading and open a PR against the repo, the build will be provisioned/attributed under my github account. Build minutes for open source should be unlimited currently.

[daniel-sudz]

for the versioning this is how it is typically handled:

1. trigger on push to wip branch
2. release based on version
3. bump version by one (ie. .wip-5 -> .wip-6)
4. push version commit back to branch

[cwensel]

fwiw, I have a full build server (on my network). everything is automated already, only I can see it for now (it was open in the past, but got bricked somehow, so I closed off the ports). that said, automating PRs would be cool.

so the real question is where to publish the artifacts.

so, we could publish them where suggested above, just need to confirm it's allowed and results in an improvement.

also, there isn't any reason to not proxy the github artifacts elsewhere. for example, https://conjars.wensel.net is read-only clone of https://www.conjars.org. Maybe we could create an action that pushes the github artifacts to the s3 bucket we use behind https://conjars.wensel.net.

sadly that will cost dollars serve out, so I'd rather find public service.

do you have an nexus/artifactory server you can proxy through?

[daniel-sudz]

personally I am not too familiar with the packaging ecosystem on java. I could look into the rules around maven-central but for now I don't see an issue with just publishing WIP there as well as normal releases

anyways, I think it's good to start with just a github-actions for running tests on PR and then move from there.

[cwensel]

ok, let's solve the first problem, getting WIPs from behind a credential wall.

I'll research the rules on Maven central, but snapshot builds (which semantically WIPs are) aren't allowed. But they do have a snapshot repo I need to look into.

if that's a dead end, i'll look at syncing the github packages with another location. the challenge is that it costs dollars to ship bytes out of s3. even 404 response errors add up, so I don't see this as a long term solution.

many corporate teams use artifactory/nexus as an internal proxy to mitigate bit supply chain security issues. this makes a case to get the artifacts published elsewhere (it's a pita to give those tools creds to access bits) as well as offers a solution (you don't need to distribute creds to get at bits).

[daniel-sudz]

side note: I've seen a decent amount of usage for sonatype https://central.sonatype.org/publish/

looks like no specific snapshot rules and seems free for OSS:
https://central.sonatype.org/publish/#individual-projects-open-source-software-repository-hosting-ossrh

After that, you can [publish](https://central.sonatype.org/publish/release/) as many components and releases under the provided Group ID and any nested Group ID values to the Central Repository as often as you like.

[daniel-sudz]

it's a bit confusing because they call themselves "Central" and use maven in the domain name but:

There is no formal relationship between Central Repository and mvnrepository.com, so we can't speak to the frequency or accuracy of their updates.

[cwensel]

so there isn't any confusion, Cascading final release are published to Maven Central (sontatype)

• https://search.maven.org/search?q=g:net.wensel

WIP release are not, because they are simply SNAPSHOT releases under a different name, and I didn't want to abuse sontatype. Policies (strict or implied) may have relaxed over the years, I'll look into it.

If I simply make a final release, the artifacts will show up in Maven Central. This is already automated.

[cwensel]

refreshed my memory a bit on this subject.

Distributing snapshot (or intermediate builds) isn't the intended purpose of Maven Central.

There is a managed repository for -SNAPSHOT releases (https://s01.oss.sonatype.org/content/repositories/snapshots/) we could try for wips, but it is not synced with Maven Central, but doesn't require creds from what I can tell. In all likelihood this will fail because wips do not use maven SNAPSHOT naming conventions. Feel free to experiment on this by tweaking the build.

That said, wip releases are intermediate releases for testing, once verified, we do an actual release to Maven Central.

Going from wip to release for a maintenance version is quick, I usually just want the features verified in a build by the community before the wip is promoted to an actual release.

For major/minor releases, this is how we test new features in flight before making the full release. we can see 100 wip builds over a period of time as features are developed. This is the case now. 4.1/4.5 isn't ready and some apis are still in flight.

As it stands, I don't see a way forward exactly. Other than make a 4.1/4.5 release, which means I need to find time to finish any outstanding work. The good news, the bits are available as a wip, in github packages, but just not super convenient due to the creds requirement.

My recommendation for that is to proxy/cache the wip releases in a local caching repository that manages the creds to github. As someone who as hosted a maven repository out of his own pocket for 10 years or so, setting up a caching proxy for maven artifacts is a good thing for reliability on the build side, and lowering costs on the serving side.

[daniel-sudz]

thanks for all the info. I agree it's not really a big deal though I might take up some experimenting with the SNAPSHOT idea if I have spare time.

The way I would approach it is per branch. So for any branch on cwensel/cascading just prepend SNAPSHOT to the branch name, set that as the project version, and publish. This would be pretty easy to implement and not require any changes. Example:

1. push happens into wip-4.5
2. artifact ID is set to wip-4.5-SNAPSHOT
3. publish happens to wip-4.5-SNAPSHOT

For PR test automation I have #86 open