Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[javascript] export neo4jcsv error #4568

Open
MichaelWalkerx opened this issue May 17, 2024 · 5 comments
Open

[javascript] export neo4jcsv error #4568

MichaelWalkerx opened this issue May 17, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@MichaelWalkerx
Copy link

MichaelWalkerx commented May 17, 2024 •
edited by max-leuthaeuser

Language: javascript
Testcase: https://gitee.com/y_project/RuoYi-Vue/tree/master/ruoyi-ui
Tool: Joern -> importCode("./ruoyi-ui") -> gen cpg.bin
exec command: joern-export --repr=all --format=neo4jcsv

image

CPG does not have dataflow overlay. Calculating.
2024-05-17 18:17:45.398 ERROR CpgPassBase: Pass io.joern.dataflowengineoss.passes.reachingdef.ReachingDefPass failed
overflowdb.SchemaViolationException: OUT edge with label REF to an adjacent METHOD is mandatory, but not defined for this METHOD_REF node with id=17841
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRefDb.referencedMethod(MethodRef.scala:870) ~[io.shiftleft.codepropertygraph-domain-classes_3-1.6.11.jar:1.6.11]
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRef.referencedMethod(MethodRef.scala:400) ~[io.shiftleft.codepropertygraph-domain-classes_3-1.6.11.jar:1.6.11]
        at io.shiftleft.codepropertygraph.generated.traversal.MethodRefTraversalExtGen$.referencedMethod$extension$$anonfun$1(MethodRef.scala:14) ~[io.shiftleft.codepropertygraph-domain-classes_3-1.6.11.jar:1.6.11]
        at scala.collection.Iterator$$anon$9.next(Iterator.scala:584) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.Iterator$$anon$10.nextCur(Iterator.scala:594) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:608) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:477) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addEdgesToCapturedIdentifiersAndParameters$1$$anonfun$2(DdgGenerator.scala:185) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addEdgesToCapturedIdentifiersAndParameters$1(DdgGenerator.scala:186) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addReachingDefEdges(DdgGenerator.scala:214) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at io.joern.dataflowengineoss.passes.reachingdef.ReachingDefPass.runOnPart(ReachingDefPass.scala:33) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at io.joern.dataflowengineoss.passes.reachingdef.ReachingDefPass.runOnPart(ReachingDefPass.scala:23) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at io.shiftleft.passes.NewStyleCpgPassBase$$anon$2.accept(CpgPass.scala:156) ~[io.shiftleft.codepropertygraph_3-1.6.11.jar:1.6.11]
        at io.shiftleft.passes.NewStyleCpgPassBase$$anon$2.accept(CpgPass.scala:155) ~[io.shiftleft.codepropertygraph_3-1.6.11.jar:1.6.11]
        at java.util.stream.ReduceOps$4ReducingSink.accept(ReduceOps.java:220) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960) ~[?:?]
        at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934) ~[?:?]
        at java.util.stream.AbstractTask.compute(AbstractTask.java:327) ~[?:?]
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:693) ~[?:?]
        at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:161) ~[io.shiftleft.codepropertygraph_3-1.6.11.jar:1.6.11]
        at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:76) ~[io.shiftleft.codepropertygraph_3-1.6.11.jar:1.6.11]
        at io.shiftleft.semanticcpg.layers.LayerCreator.runPass(LayerCreator.scala:53) ~[io.joern.semanticcpg-2.0.367.jar:2.0.367]
        at io.joern.dataflowengineoss.layers.dataflows.OssDataFlow.create$$anonfun$1(OssDataFlow.scala:31) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at io.joern.dataflowengineoss.layers.dataflows.OssDataFlow.create(OssDataFlow.scala:32) ~[io.joern.dataflowengineoss-2.0.367.jar:2.0.367]
        at io.shiftleft.semanticcpg.layers.LayerCreator.run(LayerCreator.scala:32) ~[io.joern.semanticcpg-2.0.367.jar:2.0.367]
        at io.joern.joerncli.CpgBasedTool$.addDataFlowOverlayIfNonExistent(CpgBasedTool.scala:31) ~[io.joern.joern-cli-2.0.367.jar:2.0.367]
        at io.joern.joerncli.JoernExport$.exportCpg(JoernExport.scala:103) ~[io.joern.joern-cli-2.0.367.jar:2.0.367]
        at io.joern.joerncli.JoernExport$.main$$anonfun$1$$anonfun$1(JoernExport.scala:68) ~[io.joern.joern-cli-2.0.367.jar:2.0.367]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) ~[org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.util.Using$.resource(Using.scala:262) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at io.joern.joerncli.JoernExport$.main$$anonfun$1(JoernExport.scala:69) ~[io.joern.joern-cli-2.0.367.jar:2.0.367]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15) [org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10) [org.scala-lang.scala3-library_3-3.4.1.jar:3.4.1]
        at scala.Option.foreach(Option.scala:437) [org.scala-lang.scala-library-2.13.12.jar:?]
        at io.joern.joerncli.JoernExport$.main(JoernExport.scala:69) [io.joern.joern-cli-2.0.367.jar:2.0.367]
        at io.joern.joerncli.JoernExport.main(JoernExport.scala) [io.joern.joern-cli-2.0.367.jar:2.0.367]
Caused by: java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:973) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:971) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at scala.collection.Iterator$$anon$6.next(Iterator.scala:492) ~[org.scala-lang.scala-library-2.13.12.jar:?]
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRefDb.referencedMethod(MethodRef.scala:865) ~[io.shiftleft.codepropertygraph-domain-classes_3-1.6.11.jar:1.6.11]
        ... 57 more
Exception in thread "main" overflowdb.SchemaViolationException: OUT edge with label REF to an adjacent METHOD is mandatory, but not defined for this METHOD_REF node with id=17841
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRefDb.referencedMethod(MethodRef.scala:870)
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRef.referencedMethod(MethodRef.scala:400)
        at io.shiftleft.codepropertygraph.generated.traversal.MethodRefTraversalExtGen$.referencedMethod$extension$$anonfun$1(MethodRef.scala:14)
        at scala.collection.Iterator$$anon$9.next(Iterator.scala:584)
        at scala.collection.Iterator$$anon$10.nextCur(Iterator.scala:594)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:608)
        at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:477)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addEdgesToCapturedIdentifiersAndParameters$1$$anonfun$2(DdgGenerator.scala:185)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addEdgesToCapturedIdentifiersAndParameters$1(DdgGenerator.scala:186)
        at io.joern.dataflowengineoss.passes.reachingdef.DdgGenerator.addReachingDefEdges(DdgGenerator.scala:214)
        at io.joern.dataflowengineoss.passes.reachingdef.ReachingDefPass.runOnPart(ReachingDefPass.scala:33)
        at io.joern.dataflowengineoss.passes.reachingdef.ReachingDefPass.runOnPart(ReachingDefPass.scala:23)
        at io.shiftleft.passes.NewStyleCpgPassBase$$anon$2.accept(CpgPass.scala:156)
        at io.shiftleft.passes.NewStyleCpgPassBase$$anon$2.accept(CpgPass.scala:155)
        at java.base/java.util.stream.ReduceOps$4ReducingSink.accept(ReduceOps.java:220)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
        at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
        at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
        at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:927)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:693)
        at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:161)
        at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:76)
        at io.shiftleft.semanticcpg.layers.LayerCreator.runPass(LayerCreator.scala:53)
        at io.joern.dataflowengineoss.layers.dataflows.OssDataFlow.create$$anonfun$1(OssDataFlow.scala:31)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
        at io.joern.dataflowengineoss.layers.dataflows.OssDataFlow.create(OssDataFlow.scala:32)
        at io.shiftleft.semanticcpg.layers.LayerCreator.run(LayerCreator.scala:32)
        at io.joern.joerncli.CpgBasedTool$.addDataFlowOverlayIfNonExistent(CpgBasedTool.scala:31)
        at io.joern.joerncli.JoernExport$.exportCpg(JoernExport.scala:103)
        at io.joern.joerncli.JoernExport$.main$$anonfun$1$$anonfun$1(JoernExport.scala:68)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.util.Using$.resource(Using.scala:262)
        at io.joern.joerncli.JoernExport$.main$$anonfun$1(JoernExport.scala:69)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.Option.foreach(Option.scala:437)
        at io.joern.joerncli.JoernExport$.main(JoernExport.scala:69)
        at io.joern.joerncli.JoernExport.main(JoernExport.scala)
Caused by: java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:973)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:971)
        at scala.collection.Iterator$$anon$6.next(Iterator.scala:492)
        at io.shiftleft.codepropertygraph.generated.nodes.MethodRefDb.referencedMethod(MethodRef.scala:865)
        ... 57 more
@MichaelWalkerx MichaelWalkerx added the bug Something isn't working label May 17, 2024
@max-leuthaeuser
Copy link
Contributor

Strange. Importing the cpg into joern and running:
joern> cpg.methodRef.filter(r => Try(r.method.size).isFailure).size
shown 0 results. So there are no method refs without methods coming from the frontend itself.

@johannescoetzee @DavidBakerEffendi
I am pretty sure we have seen this error already for slicing. Maybe this is some similar multi-threading issue? Not sure who of you two guys fixed it there.

@max-leuthaeuser
Copy link
Contributor

@MichaelWalkerx What joern version are you running?

@DavidBakerEffendi
Copy link
Collaborator

I usually see this error in either concurrency related bugs or low memory related runtime issues. So either could be the cause.

@MichaelWalkerx
Copy link
Author

2.0.367

@max-leuthaeuser
Copy link
Contributor

max-leuthaeuser commented May 20, 2024 •
edited

Could you try it with more memory?
export JAVA_OPTS="-Xms4G -Xmx16G"
Adjust the values for your machine.

@max-leuthaeuser max-leuthaeuser mentioned this issue May 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@max-leuthaeuser @DavidBakerEffendi @MichaelWalkerx