Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"too many executable mappings" error #2669

Open
PietroPasotti opened this issue Apr 4, 2024 · 1 comment
Open

"too many executable mappings" error #2669

PietroPasotti opened this issue Apr 4, 2024 · 1 comment

Comments

@PietroPasotti
Copy link

Describe the bug
I am running locally parca and parca-agent and I see a stack trace in parca and lots of error logs in parca-agent. I can't find traces for the python processes I'm trying to analyze (I searched by pid).
I googled the error messages but couldn't find anything useful. Can you help me figure out what's going on?

To Reproduce
run parca, parca-agent.

# parca.yaml config:
 object_storage:                                      
   bucket:                                            
     type: "FILESYSTEM"                               
     config:                                          
       directory: "./parca_data"                      
                                                                                                            
 scrape_configs:                                      
  - job_name: "parca"                                 
    scrape_interval: "10s"                            
    static_configs:                                   
      - targets: [ '127.0.0.1:7070' ]                 

run any python process (tried: ipython 8.5.0, pytest (local project, python 3.11.8)).

Expected behavior
I'd expect to see profiles for the python programs running on the host.

Screenshots or Profiles (using https://pprof.me)
n/a

Logs
From parca: (repeated multiple times)

recovered stack trace:                                                                                                                                                                         
 goroutine 232 [running]:                                                                                                                                                                      
runtime/debug.Stack()                                                                                                                                                                          
        runtime/debug/stack.go:24 +0x5e                                                                                                                                                        
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines.func1()                                                                                                                
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:107 +0x45                                                                                                                     
panic({0x4001d60?, 0xc056084270?})                                                                                                                                                             
        runtime/panic.go:914 +0x21f                                                                                                                                                            
github.com/parca-dev/parca/pkg/symbol/elfutils.(*debugInfoFile).SourceLines(0xc0284ab170, 0x243bc)                                                                                             
        github.com/parca-dev/parca/pkg/symbol/elfutils/debuginfofile.go:120 +0x7a8                                                                                                             
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines(0xc051b00500?, 0xc056027680?)                                                                                          
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:112 +0x6c                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).pcToLines(0xc0016dfad0, {0x60b65a0, 0xc02a826d40}, {0xc0551d3050, 0x28}, 0x0?)                                                         
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:574 +0x53                                                                                                                      
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolizeLocationsForMapping(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0xc051ee3880, {0xc051ee6de0, 0x3, 0xc051ee3680?})                
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:517 +0x1afb                                                                                                                    
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Symbolize(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, {0xc000efe600, 0x32f, 0x7?})                                                        
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:297 +0x971                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).runSymbolizationCycle(0xc0016dfad0, {0x60b5bf0, 0xc00185f890})                                                                         
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:221 +0x152                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1()                                                                                                                            
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:190 +0xf1                                                                                                                      
github.com/parca-dev/parca/pkg/runutil.Repeat(0xc001222040?, 0xc0001822a0, 0xc007e15dd8)                                                                                                       
        github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x82                                                                                                                             
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0x3f87f00?)                                                                               
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:188 +0x73                                                                                                                      
github.com/parca-dev/parca/pkg/parca.Run.func5.1({0x60b5bf0?, 0xc00185f890?})                                                                                                                  
        github.com/parca-dev/parca/pkg/parca/parca.go:531 +0x3d                                                                                                                                
runtime/pprof.Do({0x60b5c28?, 0xc0012c4e60?}, {{0xc001222040?, 0x0?, 0x0?}}, 0xc00187af40)                                                                                                     
        runtime/pprof/runtime.go:51 +0x9d                                                                                                                                                      
github.com/parca-dev/parca/pkg/parca.Run.func5()                                                                                                                                               
        github.com/parca-dev/parca/pkg/parca/parca.go:530 +0x1ef                                                                                                                               
github.com/oklog/run.(*Group).Run.func1({0xc0013374c0?, 0xc0013374e0?})                                                                                                                        
        github.com/oklog/[email protected]/group.go:38 +0x29                                                                                                                                          
created by github.com/oklog/run.(*Group).Run in goroutine 1                                                                                                                                    
        github.com/oklog/[email protected]/group.go:37 +0x67                                                                                                     

From parca-agent: (repeated many times)

level=warn name=parca-agent ts=2024-04-04T12:48:18.16188908Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"            
level=warn name=parca-agent ts=2024-04-04T12:48:19.081906622Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:21.172132294Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:21.173922614Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:22.5785825Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"             
level=warn name=parca-agent ts=2024-04-04T12:48:22.879245819Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:24.995136311Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=error name=parca-agent ts=2024-04-04T12:48:25.245026903Z caller=maps.go:1279 component=cpu_profiler component=bpf_maps msg="addUnwindTableForProcess failed" err="setUnwindTableForMapping for executable /home/me/lib/python3.10/site-packages/pydantic_core/_pydantic_core.cpython-310-x86_64-linux-gnu.so starting at 0x7a940524b000 failed: not enough profiling rounds with this unwind info"                                      
level=warn name=parca-agent ts=2024-04-04T12:48:25.991695868Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:25.995384796Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:26.493269298Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:28.751652154Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           

Software (please complete the following information):
parca, version 0.21.0 (commit: 089e78068a1c4f8773b1d5a8bac6645c6414fdfd)
parca-agent, version 0.30.0 (commit: da59a7c, date: 2024-03-04T20:55:59Z), arch: amd64

Workload (please complete the following information):
pytest ./some-local-project

Environment (please complete the following information):
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"

  • 6.5.0-26-generic
  • x86_64
@brancz
Copy link
Member

brancz commented Apr 9, 2024

I opened #2675 to improve the error message so we understand better how many mappings would be enough.

I think the best resolution for this would be if we could make the max number of mappings configurable.

Regarding the symbolization, we actually just started revamping the symbolizer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants