About the jq's release process (Was: Is jq is still alive/maintained ?)

[tst2005]

Hello,

jq is a already powerfull enough for most of use.
it does not need new release too often but ...
There are lot of PR for bugfixes and interesting improvements.

is @stedolan still maintaining jq ?
if it is a "dead" project, Am I alone to think about a fork ?

Regards,

[carlsmedstad]

Also wondering this. It would be a shame if it stopped being developed and maintained. Maintainers, is there anyway to help you out?

[wtlangford]

Hey, so I'm still around- the main issue is that I've been super busy with work and dealing with life this last year that it's been especially hard to find the time/energy to devote to jq.

We'd definitely appreciate help-
I've got a couple small changes floating around that I need to finish up which should improve some build behaviors (I think I can make the CI builds more reliable- they keep breaking. Should also be able to fix some things around the math functions that got brought up "recently"). I'll also try and dig up that list of things that needed finishing in order to cut the next release and see if I can set up some issues for it and a milestone so that it's clear what needs doing. (Depending on what's left on that, I might just say we should cut a release and schedule myself some time to do it).

Beyond that, I need to go review all the open PRs- I know I've seen some good stuff come through my email- and also try to clean up the issue tracker some.

[tst2005]

Thanks for your answers.

@carlsmedstad : IMHO, I don't think it is a shame if @stedolan can not or don't want maintain jq anymore. I prefer saying a big thank for the work already done.

Today
I see the stedolan's jq repository has 19.3k stars, 1.1k fork, 121 PR, 574 issues.
121/1100 --> ~10% of forks are PR
574/1100 --> ~50% of forks equal to an issue (bug/ask/feature request/...)

There different solutions to get a projet maintained, :

• the original author is still here, he can do stuff to transfert the role to another guy/community.
• the original author is not here. The repository is forked to a new "official/maintained" one, it is difficult to make people create issue/PR on the new repo because all current search result still anwer the original repository.

For now I don't read people answering "I forked and I can maintain jq",
@wtlangford you are the closest one but you already told us you are busy.

I'm in the same case. I want a maintained jq, I can contribute but I don't have enough time to be the only one to maintain jq.

When nobody has the time to maintain the projet, we can try to make a github "organisation" like "jq-community" or "jq-maintainers". It imply to work together (it is not always easy).

My personnal goal and possible contribution is targeted on jq builtin

I'm currently using jq 1.5, the only interesting thing I take from jq 1.6 is some builtin (made in jq language it self) then I took them and load on jq 1.5 before using them.

The most work I'm doing around jq is provide a set of usefull functions/samples for my colleages and myself.
Some of them could be jq builtin, some other not.
jq is powerfull but not always easy to use.

I want more sample in documentation for builtin and more wiki cookbook) for functions that is not built in but interesting enough to exist.

For me, there are a few number of missing feature in jq.
The only main one for me is the ability to check if a function is already defined or not.
For example: It could be usefull to be able to define a math pow function (made in jq language), only if a "C native" pow not already available.

[itchyny]

I would like to help maintaining this project if you could invite me as a collaborator. Maybe @pkoppstein would be willing to help, too. I have already submitted some pull request, including the fix for the CI (#2141, #2193). Based on my experience of gojq, I believe I have throughout knowledge of jq filters. But I'm afraid there are many difficulties in this project though; changing the behaviors will likely to break some scripts in the world, even if it is likely broken behaviors (like leaf_paths misjudging falcy values #1163 and index returning bytes offsets #1430). But still I can help reviewing document fixes or closing questions.

[carlsmedstad]

@wtlangford Fully understand that, really appreciate the work you do. I'm not too familiar with the project yet but I'll try to get to it this weekend and see where I could help out.

@tst2005 Sorry, I did not intend to come off as negative or demanding. I just wanted to get across that I appreciate the project and would like to help.

[wader]

@wtlangford Thanks for all hard work. Im also willing to help out with review, documenting, testing and whatnot

[tst2005]

I create a https://github.com/jqmaintainers organisation,
@wader @carlsmedstad @itchyny @wtlangford I invited you all to join.
EDIT: I also invited @leonid-s-usov @nicowilliams @pkoppstein, any help is welcome ;)

[leonid-s-usov]

Hi!

I feel bad that at some point I fell out of the work that I've started on a couple of core developments here. Back then I had some spare period between jobs but since I've entered my current position I haven't been having any spare time/brain to continue working on this.

I am very sad to see how this amazing project is struggling currently and I would encourage @nicowilliams and @wtlangford to seriously consider picking a responsible maintainer who will be able to put the required effort into moving this forward.

Personally, I am still planning to find some time to finalize my work on a very promising feature that Nico has started. It is going to open the way to extending JQ with custom filters and producers, like fetching stuff from APIs, or running queries against DBs from within the jq script.

[davidfetter]

That maintenance is landing on one person who may or may not actually have the time because they have other obligations such as a day job is precisely the problem. Making a GH organization and seeing to it that we have enough people who can actually do the work would go a very long way in the right direction.

[pkoppstein]

@tst2005 - Some thoughts in response to your post:

1. jq has had pow/2 since 1.5.

2. Since the (official) Cookbook has become a collection of short essays rather than a "library" of ready-to-go functions, one thought would be to create a new page on the wiki for code snippets. Perhaps "library" or "snippets" or "supplemental builtins" would be a suitable title?

3. There are currently 475 jq entries (mostly worked examples) at rosettacode.org (http://rosettacode.org/wiki/Category:Jq).

[itchyny]

Decision such as migrating to an organization should happen after deliberation of current maintainers.

[tst2005]

@pkoppstein

1. jq has had pow/2 since 1.5.

I know it is was just a simple example.
It can be use to make compat from 1.5 to 1.6 for sql-like function.

2. Since the (official) Cookbook has become a collection of short essays rather than a "library" of ready-to-go functions, one thought would be to create a new page on the wiki for code snippets.  Perhaps "library" or "snippets" or "supplemental builtins" would be a suitable title?

For me there are 3 kind of functions:

• very usefull : must be included (embedded as built-in)
• usefull : we hesitate to put it as built-in. then put them in a maintained extended.jq file, easily loadable/usable
• misc : there are stuff that can be usefull but in very limited case, it can be placed in cookbook.

3. There are currently 475 jq entries (mostly worked examples) at rosettacode.org (http://rosettacode.org/wiki/Category:Jq).

I'm not sure to understand. What you wanna do with that ?

[tst2005]

@itchyny

Decision such as migrating to an organization should happen after deliberation of current maintainers.

He didn't react/commit for years. I supposed he does care of jq anymore.
I will be happy if he answer us !

[wtlangford]

@itchyny

Decision such as migrating to an organization should happen after deliberation of current maintainers.

He didn't react/commit for years. I supposed he does care of jq anymore.
I will be happy if he answer us !

So these days, @nicowilliams and I are the two maintainers. I'll chat with him about it and see what he thinks

For me, there are a few number of missing feature in jq.
The only main one for me is the ability to check if a function is already defined or not.

While slightly off-topic for this thread, I do want to point out that this isn't really possible, due to how jq's internal linker works. When we're parsing your code, we just note that you referred to a symbol by name. Later, we do a pass through to try and link your symbols to already defined things. If we can't find a symbol by that name, linking fails. So there's not really a way to do that, without introducing some kind of preprocessor, and that's not something I'm particularly enamored of.

It can be use to make compat from 1.5 to 1.6 for sql-like function.

So one of the nice things about jq is that most of our standard library is defined as jq code (https://github.com/stedolan/jq/blob/master/src/builtin.jq). So as long as it doesn't actually depend on a new language feature, you can just copy over the needed library functions.

That said, per my comments earlier, I'm going to spend some time today doing some cleaning/organizing of issues/PRs so that we can see what's left before a 1.7 release.

[pkoppstein]

@tst2005 -

1. I mentioned rosettacode.org because you wrote:

I want more sample in documentation for builtin and more wiki cookbook) for functions that is not built in but interesting enough to exist.

2. Regarding builtin.jq vs extended.jq vs misc.jq - it takes quite a lot of joint discussion and time before anything new makes its way into builtin.jq, and whether an extended.jq will ever make it into the official distribution has not yet even been discussed to my knowledge, so it seems to me that one or perhaps two new pages on the wiki would be helpful for experimentation and discussion, as well as perhaps being quite useful.

3. Regarding your comment:

He didn't react/commit for years.

You are perhaps referring to the original creator of jq, but there have been other official maintainers, and of course at least one of them (thank you @wtlangford!) is participating in this thread.

[wtlangford]

Regarding builtin.jq vs extended.jq vs misc.jq - it takes quite a lot of joint discussion and time before anything new makes its way into builtin.jq, and whether an extended.jq will ever make it into the official distribution has not yet even been discussed to my knowledge, so it seems to me that one or perhaps two new pages on the wiki would be helpful for experimentation and discussion, as well as perhaps being quite useful.

We have actually discussed it some, but we're not sure how we want to handle it- anything like that would be a breaking change for a lot of existing scripts, and that's something we'd like to avoid. The main reason jq-1.6 is so much slower than 1.5 is due to the sheer number of builtins we added to it and the cost that incurs at load-time. Splitting less-commonly needed builtins out into builtin modules would go a long way towards helping (as we wouldn't load those modules unless you ask for them).

But, as I said- that's going to be breaking behavior and I'm hesitant to commit to it. Beyond that, we had a great contribution back in 2019 (yes, I know our release schedule is slow) that significantly improved startup times back to approximately the performance of jq-1.5, so it's a bit less pressing to my mind.

[kasperk81]

That maintenance is landing on one person who may or may not actually have the time because they have other obligations such as a day job is precisely the problem. Making a GH organization and seeing to it that we have enough people who can actually do the work would go a very long way in the right direction.

This is a solid point, it does not mean to blame any one individual just a fact about the nature of things in open source, people move on, lose interest, life happens and have (real) morning jobs. An organization with multiple entities makes much sense for project like jq to evolve. Lets face it, it is simply not one individual's job to maintain this huge popular project at this point.

[chb0github]

Eventually all great projects outgrow their original maintainers (hello, Linux). I definitely would encourage whomever still owns this to delegate and allow the community to grow.

I could see a whole community with several branches springing up. I, for one, would love to see a whole repo dedicated to add-on functions that can be installed.

Please consider growing the community and releasing soon

[psacawa]

Can I second those who ask for a new release from the new maintainers? jq has had optional else clause in if statement since 4f6045a in Feb 2020. Since then it doesn't appear in any release. It's pretty strange that such a useful feature is locked away in the master branch where the people who install from package repository can't use it.

[chb0github]

To the owner of this repo: If you simply don't have the time to even delegate, then donate it to something like the apache foundation or the eclipse foundation, that has a great record of running open source projects

Is the owner even still alive? @stedolan - he hasn't made a single comment on this thread

[cheister]

@wtlangford Any updates on doing a release?

[chb0github]

He's dead. We need a fork for someone who can maintain it

[vintnes]

This comment is a rant; accept my apologies and skip it.

@tst2005 @chb0github

Fork it then. This was a complicated project before the portability concerns and the massive adoption. It's a programming language that most of its users don't know is a programming language. It's a functional shell-oriented streaming DSL used by people who write AWS docs and people who think Python is bloat. Its maintenance requirements are unique. Anyone who's capable of wrangling libjq is also capable of making a quarter million a year bumping the version control on some embedded firmware.

I'm on record begging for a new release but if you look at the project history the schedule isn't weird and the build isn't expensive. Emailing everyone subscribed to this repo the same entitled comments over and over does not make anything happen faster. There are some tantalizing features in development but I, for one, would rather see the scope protected than the growth rate explode.

@wtlangford Sorry for contributing to the trash fire

[chb0github]

I depart. Clearly nothing productive here.

[tst2005]

For a more productive comment...

@wtlangford
I supposed you are little busy and do what you can with the free time you have (like most of us!)

1. the release process

Can you consider to make Release candidate ? jq 1.7rc1, 1.7rc2, 1.7rc3 ... before the final jq 1.7 release ?
It could be a way to get more feed back about feature, benchmark, etc.
If you don't have time, you should maybe delegate the release candidate to others peoples. and keep the final release for your own.
You probably be the one to be able to make a clean official release, but we can try to make some release candidate (or "beta" ?) to help you to move to the next release and test what it should be included or not.

2. multiple version support

I'm using jq over the debian package distribution currently I'm using jq 1.5 I'm loading some interesting jq 1.6 built-in function when I need it.
I'm not sure but I read somebody told jq 1.6 is less performant than jq 1.5
If it's true, how to deal with this ?
Can we consider to maintain multiple branches ? and make backports release, include recent features in an older version
My personnal need should only be a jq 1.5 with the jq 1.6 built-in, that can be named jq 1.5.1 or jq 1.5.0+bpo1 ...
I just launch the idea. I'm don't know if it is a really good way to do.

Regards,

[jpmckinney]

FWIW, I'm trying jaq and it is faster and does the complex queries I have (I haven't checked whether 100% compliant): https://github.com/01mf02/jaq

[pkoppstein]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be.
The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening.

By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

[eitsupi]

This issue mey be outdated? (This repo is now jqlang/jq)

[jpmckinney]

Looks like that change happened within the last 24 or 48 hours, yes.

Edit: #2594 (comment)

[kloczek]

So looks like whole repo with all data/metadata has been transferred to new maintainer 🤔

[xinthose]

He's dead. We need a fork for someone who can maintain it

he hasn't responded or made a commit in over two years, so you could be right

[nicowilliams]

He's dead. We need a fork for someone who can maintain it

he hasn't responded or made a commit in over two years, so you could be right

We have:

• a new home for jq (jqlang/jq) -- a new org, so that we're not dependent on just one person
• new maintainers
• new org owners and admins
• work being done towards making a release

Yes, the old maintainers (myself included) went AWOL, but we're not dead. You can thank @owenthereal for pushing to do this, and @stedolan for transferring the old stedolan/jq repository to the new jqlang org.

[nicowilliams]

So looks like whole repo with all data/metadata has been transferred to new maintainer 🤔

Same maintainers but with more new maintainers, because we the old maintainers were not responsive.

[nicowilliams]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be. The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening.

By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

I'm bored of C -- it's a very dangerous language. I think Rust is a pretty good choice of a language for a rewrite. Just looking at the README for jaq I see some issues that should be addressed, but nothing serious, and if jaq can grow up to replace this implementation of jq, I think that'd be fantastic.

[jpmckinney]

I think there will always be a place for jq. From jaq's readme:

jaq currently does not aim to support several features of jq, such as:

• Paths
• Modules
• Dates
• SQL-style operators
• Streaming

I assume jaq is fast not only thanks to the author's efforts to make the implemented features faster, but also thanks to the architecture being simpler by excluding some features. It might not be possible to maintain that performance and implement all of jq's features (unless the slower features are made to use an entirely different code path, but then why bother if jq already exists for those needs).

[liquidaty]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq; either way I only found the question raised but not answered

[davidfetter]

Regarding jaq, @jpmckinney wrote:

I haven't checked whether 100% compliant

jaq is very impressive in many ways, but although it is substantially compliant, it is not, and does not strive to be. The documentation is in fact very up-front about this, so the above-quoted comment is a bit disheartening.
By contrast, gojq is substantially compliant with jq except mainly for bugs and certain "undocumented" features (e.g. _nwise/1) and "implementation" matters, notably (on the plus side) integer precision and (on the negative side) memory management.

I'm bored of C -- it's a very dangerous language. I think Rust is a pretty good choice of a language for a rewrite. Just looking at the README for jaq I see some issues that should be addressed, but nothing serious, and if jaq can grow up to replace this implementation of jq, I think that'd be fantastic.

The latest organizational Rust issue makes that seem like a bad bet, however technically sweet the technology may be.

I get that when you've been doing a lot of C, those B&D languages can start to look great, but it really is a "the grass is greener on the other side of the fence" situation, or at least it has been over the past few decades.

[wader]

Being jq compatible would probably also include re-implement all of jq's CLI argument and behaviours (input handling, exit codes etc) and also in some cases implement some behaviours that might be considered bugs, ex 1+3 as $a | ($a*2) is 8 with jaq, 7 with jq. But i do think it's possible but would be a lot of work.

BTW the author or jaq @01mf02 has written a paper about it https://arxiv.org/abs/2302.10576

[flying-sheep]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq

One of the reasons Rust became popular is that it was easy to compile to WASM from very early on. Rust remains one of the best choices to do what you want, I bet one could very quickly set up a small wrapper around jaq that exposes it to JS, e.g. using https://github.com/rustwasm/wasm-pack

The latest organizational Rust issue makes that seem like a bad bet, however technically sweet the technology may be.

Does it though? I’m so bored by any predictable outbreak of internet drama being FUDded into the doom of humankind. Rust is used in the Windows kernel. It’s not going away.

[eitsupi]

Obviously the original problem with this issue has been resolved, so why not create another issue about the rewrite to Rust and close this issue?

[itchyny]

Although we're moving forward on the new jqlang org with new maintainers, we haven't go through
the entire releasing process, so let's keep this open.

[tst2005]

Although we're moving forward on the new jqlang org with new maintainers, we haven't go through the entire releasing process, so let's keep this open.

The survive of jq seems now more assured than at the beginning of this thread ;-)

[01mf02]

I would happily use jaq, if it can support wasm targets (more specifically, can be linked as a library into a wasm module). Maybe this is more of a general Rust matter than jaq; either way I only found the question raised but not answered

@liquidaty, the question you have referenced was about having jaq as a "WebAssembly library wrapped in JS on npm". I have never done the "npm" part, but I have already compiled another Rust program of mine to WebAssembly and used it from JS. So this part is definitely doable, also for jaq. I would be really happy if somebody would try it, because having something like jqplay.org for jaq would be quite cool. In particular, it would allow processing all data on the client instead of sending it to the server.

[colindean]

I wish I could react to the close notification so this comment will have to do. Great work, new maintainer team!

[nicowilliams]

About the jq's release process (Was: Is jq is still alive/maintained ?)

Yes! jq has been revived, and now is indeed alive and being maintained again. You can thank... a lot of people for this, including those who asked what was up and even threatened a fork.

[tst2005]

After more than 2 years, I'm really happy to see this issue that can now be closed without hesitation ;-)

[nicowilliams]

After more than 2 years, I'm really happy to see this issue that can now be closed without hesitation ;-)

Just so you know, your efforts to fork jq helped revive jq. Thanks!

[matei-dragu]

Dev Team - Thank you very much! Amazing job!

[jbrains]

This is really good timing for me. I wanted modules last week and I'm using jaq, so suddenly I wanted to use jq again. :)