Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allocated maps are ignored #241

Open
christianhadler opened this issue Feb 6, 2024 · 0 comments
Open

Allocated maps are ignored #241

christianhadler opened this issue Feb 6, 2024 · 0 comments

Comments

@christianhadler
Copy link

I'm observing huge performance hits when collecting values into a an object by using reduce:

gojq:

$ time gojq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null

real    0m49.581s
user    0m0.015s
sys     0m0.031s

jq:

$ time jq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null

real    0m0.156s
user    0m0.109s
sys     0m0.062s

It seems that a new map is allocated (and filled) on every iteration:

package main

import (
	"os"
	"runtime/pprof"

	"github.com/itchyny/gojq"
)

func main() {
	fd, err := os.Create("profile.pprof")
	if err != nil {
		panic(err)
	}
	defer fd.Close()

	pprof.StartCPUProfile(fd)
	defer pprof.StopCPUProfile()

	query, err := gojq.Parse(`[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})`)
	if err != nil {
		panic(err)
	}

	code, err := gojq.Compile(query)
	if err != nil {
		panic(err)
	}

	iter := code.Run(nil)
	for {
		v, ok := iter.Next()
		if !ok {
			break
		}
		if err, ok := v.(error); ok {
			panic(err)
		}
	}
}
(pprof) list gojq.updateObject
Total: 78.57s
ROUTINE ======================== github.com/itchyny/gojq.updateObject in C:\Users\chrha\go\pkg\mod\github.com\itchyny\[email protected]\func.go
     1.75s     34.67s (flat, cum) 44.13% of Total
         .          .   1604:func updateObject(v map[string]any, k string, path []any, n any, a allocator) (any, error) {
         .          .   1605:   x, ok := v[k]
         .          .   1606:   if !ok && n == struct{}{} {
         .          .   1607:           return v, nil
         .          .   1608:   }
         .          .   1609:   u, err := update(x, path, n, a)
         .          .   1610:   if err != nil {
         .          .   1611:           return nil, err
         .          .   1612:   }
         .          .   1613:   if a.allocated(v) {
         .          .   1614:           v[k] = u
         .          .   1615:           return v, nil
         .          .   1616:   }
         .      830ms   1617:   w := a.makeObject(len(v) + 1)
     770ms      6.66s   1618:   for k, v := range v {
     980ms     27.17s   1619:           w[k] = v
         .          .   1620:   }
         .       10ms   1621:   w[k] = u
         .          .   1622:   return w, nil
         .          .   1623:}
         .          .   1624:
         .          .   1625:func updateArrayIndex(v []any, i int, path []any, n any, a allocator) (any, error) {
         .          .   1626:   var x any
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant