Uploaded image for project: 'Erlang/OTP'
  1. Erlang/OTP
  2. ERL-1453

process_main at beam_hot.h Segmentation fault (core dumped)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 19.3
    • Fix Version/s: None
    • Component/s: erts
    • Labels:

      Description

      Hello, We have experienced twice segfaults in the last two months. but we still dont't know how it happens even if after We inspected a recent core dump.

      PS: After we migration application from Physical machine to Cloud server, up to now,it hasn't happened.

       

       It's the analyze information:

       

      gdb /usr/local/erlang_19.3/erts-8.3/bin/beam.smp -core core-7_scheduler-19356-1606315051
      GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
      Copyright (C) 2010 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /usr/local/erlang_19.3/erts-8.3/bin/beam.smp...done.
      [New Thread 19890]
      [New Thread 19906]
      [New Thread 19924]
      [New Thread 19718]
      [New Thread 19914]
      [New Thread 19935]
      [New Thread 19921]
      [New Thread 19654]
      [New Thread 19856]
      [New Thread 19936]
      [New Thread 19715]
      [New Thread 19922]
      [New Thread 19598]
      [New Thread 19937]
      [New Thread 19784]
      [New Thread 19904]
      [New Thread 19716]
      [New Thread 19920]
      [New Thread 19917]
      [New Thread 19781]
      [New Thread 19916]
      [New Thread 19356]
      [New Thread 19918]
      [New Thread 19714]
      [New Thread 19928]
      [New Thread 19934]
      [New Thread 19717]
      [New Thread 19858]
      [New Thread 19855]
      [New Thread 19926]
      [New Thread 19938]
      [New Thread 19713]
      [New Thread 19432]
      [New Thread 19930]
      [New Thread 19851]
      [New Thread 19847]
      [New Thread 19711]
      [New Thread 19861]
      Missing separate debuginfo for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so
      Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/41/c5f166597dbf7614a52873d44a8896eb5b9019
      Missing separate debuginfo for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so
      Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0c/337dc399c3b1b393d15def315e4f0855e79cec
      Missing separate debuginfo for /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so
      Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/0a805b2fc1a803ee953e013bb6888f81e4847e
      Missing separate debuginfo for 
      Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/c78bce9663e631f5be6d8fb0a5843ec4a51e7a
      Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libutil.so.1
      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libtinfo.so.5
      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so...done.
      Loaded symbols for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so
      Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libcrypto.so.10
      Reading symbols from //lib64/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for //lib64/libz.so.1
      Reading symbols from /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so...done.
      Loaded symbols for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so
      Reading symbols from /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so...done.
      Loaded symbols for /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so
      Core was generated by `/usr/local/erlang_19.3/erts-8.3/bin/beam.smp -K true -P 500000 -t 10485760 -fnu'.
      Program terminated with signal 11, Segmentation fault.
      #0  0x000000000043ce7f in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:466
      466 x86_64-unknown-linux-gnu/opt/smp/beam_hot.h: No such file or directory.
          in x86_64-unknown-linux-gnu/opt/smp/beam_hot.h
      Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 openssl-1.0.1e-57.el6.x86_64 zlib-1.2.3-29.el6.x86_64
      (gdb) bt
      #0  0x000000000043ce7f in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:466
      #1  0x00000000004e0cf5 in sched_thread_func (vesdp=0x7f7184d099c0) at beam/erl_process.c:8206
      #2  0x000000000062ec02 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:114
      #3  0x0000003812e07aa1 in start_thread () from /lib64/libpthread.so.0
      #4  0x00000038126e8aad in clone () from /lib64/libc.so.6
      (gdb) source etp-commands.in
      %---------------------------------------------------------------------------
      % Use etp-help for a command overview and general help.
      %
      % To use the Erlang support module, the environment variable ROOTDIR
      % must be set to the toplevel installation directory of Erlang/OTP,
      % so the etp-commands file becomes:
      %     $ROOTDIR/erts/etc/unix/etp-commands
      % Also, erl and erlc must be in the path.
      %---------------------------------------------------------------------------
      etp-set-max-depth 20
      etp-set-max-string-length 100
      --------------- System Information ---------------
      OTP release: 19
      ERTS version: 8.3
      Compile date: Thu Aug  2 02:46:48 2018
      Arch: x86_64-unknown-linux-gnu
      Endianness: Little
      Word size: 64-bit
      HiPE support: yes
      SMP support: yes
      Thread support: yes
      Kernel poll: Supported and used
      Debug compiled: no
      Lock checking: no
      Lock counting: no
      Node name: 'xxxx'
      Number of schedulers: 24
      Number of async-threads: 10
      --------------------------------------------------
      (gdb) etp-process-info c_p
        Pid: <0.1061.0>
        State: on-heap-msgq | trapping-exit | running | active | prq-prio-normal | usr-prio-normal | act-prio-normal
        Registered name: rank_105
        Current function: rank_cmp:cmp0/2
        CP: #Cp<lists:rfmerge2_2/5+0xa0>
        I: #Cp<rank_cmp:cmp0/2+0x28>
        Heap size: 196650
        Old-heap size: 999631
        Mbuf size: 0
        Msgq len: 0 (inner=0, outer=0)
        Parent: <0.1049.0>
        Pointer: (Process *) 0x7f71480c2650
      (gdb) etp-stacktrace c_p
      % Stacktrace (37): #Cp<lists:rfmerge2_2/5+0xa0>.
      #Cp<lists:rfmergel/4+0x100>.
      #Cp<serv_rank_work:sort/3+0xc0>.
      #Cp<rank_handle:set_1/2+0x378>.
      #Cp<0xf961a310>.
      #Cp<serv_rank_work:'-handle_cast/2-fun-0-'/2+0x70>.
      #Cp<serv_rank_work:handle_cast/2+0x2b0>.
      #Cp<gen_server:try_dispatch/4+0x80>.
      #Cp<gen_server:handle_msg/5+0x6a0>.
      #Cp<proc_lib:init_p_do_apply/3+0x58>.
      #Cp<terminate process normally>.
      (gdb) etp-stackdump c_p
      % Stackdump (37): #Cp<lists:rfmerge2_2/5+0xa0>.
      #Cp<lists:rfmergel/4+0x100>.
      [{t_rank,114,{26415122618390,5010},26415122618390,618,29056,[5010,26415123051068,0],1601296765,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122618835,5010},26415122618835,617,29173,[5010,26415122720131,0],1604110584,[],#{Keys:{cardlv} Values:{47}}},{t_rank,114,{26415122624460,5010},26415122624460,616,29399,[5010,26415122862220,0],1602336667,[],#{Keys:{cardlv} Values:{63}}},{t_rank,114,{26415122619939,5010},26415122619939,611,29824,[5010,26415122814090,0],1601301808,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122613947,5010},26415122613947,609,29914,[5010,26415122953862,0],1602894861,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122615579,5010},26415122615579,603,29914,[5010,26415122671508,0],1601357472,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122623169,5010},26415122623169,602,29914,[5010,26415122822305,0],1601110803,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612247,5010},26415122612247,601,29914,[5010,26415122616860,0],1601105527,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612851,5010},26415122612851,600,29914,[5010,26415122625024,0],1601048676,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612835,5010},26415122612835,593,30463,[5010,26415122622197,0],1601971140,[],...},{t_rank,114,{26415122623537,5010},26415122623537,592,30463,[5010,26415122901147,0],1601966108,...},{t_rank,114,{26415122621609,5010},26415122621609,586,30484,[5010,26415123122717,0],...},{t_rank,114,{26415122623858,5010},26415122623858,585,30484,...},{t_rank,114,{26415122621507,5010},26415122621507,584,...},{t_rank,114,{26415122617832,5010},26415122617832,...},{t_rank,114,{26415122620384,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
      #Header7<0x7f7188132320>.
      [{t_rank,114,{26415122612527,5010},26415122612527,636,28747,[5010,26415122624701,0],1601788284,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122614702,5010},26415122614702,650,28508,[5010,26415122661854,0],1601741550,[],#{Keys:{cardlv} Values:{51}}},{t_rank,114,{26415122613255,5010},26415122613255,657,28198,[5010,26415122699977,0],1601599280,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122622439,5010},26415122622439,667,27965,[5010,26415122802651,0],1601386883,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122616871,5010},26415122616871,682,26585,[5010,26415122687473,0],1605079561,[],#{Keys:{cardlv} Values:{37}}},{t_rank,114,{26415122617778,5010},26415122617778,696,25022,[5010,26415123457430,0],1604309137,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122620386,5010},26415122620386,697,25022,[5010,26415123437455,0],1604318705,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122621655,5010},26415122621655,698,25022,[5010,26415123017419,0],1605204864,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122619745,5010},26415122619745,713,24947,[5010,26415123171428,0],1601721137,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122616726,5010},26415122616726,721,24947,[5010,26415123120864,0],1601964666,[],...},{t_rank,114,{26415122624723,5010},26415122624723,730,24947,[5010,26415123274310,0],1602598948,...},{t_rank,114,{26415122625177,5010},26415122625177,731,24947,[5010,26415123242444,0],...},{t_rank,114,{26415122615992,5010},26415122615992,745,24947,...},{t_rank,114,{26415122614158,5010},26415122614158,746,...},{t_rank,114,{26415122626524,5010},26415122626524,...},{t_rank,114,{26415122624886,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
      {t_rank,114,{26415122615452,5010},26415122615452,0,28921,[5010,26415123509209,0],1606302093,[],#{Keys:{cardlv} Values:{1}}}.
      [{t_rank,114,{26415122621785,5010},26415122621785,625,28846,[5010,26415122975186,0],1603119085,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122624394,5010},26415122624394,633,28747,[5010,26415122866146,0],1601205815,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122617794,5010},26415122617794,637,28747,[5010,26415123281000,0],1601981880,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122620931,5010},26415122620931,0,28682,[5010,26415123524034,0],1606125427,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122612690,5010},26415122612690,0,22295,[5010,26415123448883,0],1605939923,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122622564,5010},26415122622564,873,22220,[5010,26415123001022,0],1601657784,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122620996,5010},26415122620996,874,22220,[5010,26415123268060,0],1601660542,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122625649,5010},26415122625649,881,22220,[5010,26415123295399,0],1601909090,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122625165,5010},26415122625165,889,22220,[5010,26415123356718,0],1602277471,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122622085,5010},26415122622085,906,22220,[5010,26415123427782,0],1603355686,[],...},{t_rank,114,{26415122613823,5010},26415122613823,913,22220,[5010,26415123383544,0],1603613109,...},{t_rank,114,{26415122622845,5010},26415122622845,921,22220,[5010,26415123462639,0],...},{t_rank,114,{26415122618131,5010},26415122618131,923,22220,...},{t_rank,114,{26415122626415,5010},26415122626415,929,...},{t_rank,114,{26415122618845,5010},26415122618845,...},{t_rank,114,{26415122626689,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
      {t_rank,114,{26415122625184,5010},26415122625184,619,28921,[5010,26415123518673,0],Cannot access memory at address 0xe0000001d000170
      

        Attachments

        1. ErtsSchedulerData.gdb
          10 kB
        2. etp-heapdump.gdb
          1.55 MB
        3. etp-processes.gdb
          437 kB
        4. etp-process-info.gdb
          2 kB
        5. etp-stackdump.gdb
          39 kB
        6. PointerProcess.gdb
          5 kB

          Activity

            People

            Assignee:
            otp_team_vm Team VM
            Reporter:
            zboyzb zboyzb
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: