use_grpc throws "Segmentation fault" error after upgrade from 0.12.4 to 0.12.5

[jagdish-176]

I'm using the below configuration for plugin.
this is working with 0.12.4 version but not the latest 0.12.5

    @type google_cloud
    ...
    use_metadata_service false
    buffer_type file
    buffer_path 'C:\xxx\other_logs'
    buffer_chunk_limit 5120KB
    flush_interval 5s
    disable_retry_limit false
    retry_limit 3
    retry_wait 10
    max_retry_wait 300
    num_threads 8
    use_grpc true
    partial_success true

Errors as below:

C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789: [BUG] Segmentation fault
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x64-mingw32]

-- Control frame information -----------------------------------------------
c:0023 p:---- s:0145 e:000144 CFUNC  :new
c:0022 p:0155 s:0141 e:000140 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789
c:0021 p:0008 s:0127 e:000126 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:638
c:0020 p:0004 s:0123 e:000122 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/compat/call_super_mixin.rb:42
c:0019 p:0011 s:0119 e:000118 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:200
c:0018 p:0007 s:0115 e:000114 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:189 [FINISH]
c:0017 p:---- s:0111 e:000110 CFUNC  :each
c:0016 p:0069 s:0107 e:000106 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188 [FINISH]
c:0015 p:---- s:0100 e:000099 CFUNC  :each
c:0014 p:0049 s:0096 e:000095 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175
c:0013 p:0006 s:0088 e:000087 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:199
c:0012 p:0005 s:0084 e:000083 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:248
c:0011 p:0035 s:0080 e:000079 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:147
c:0010 p:0120 s:0075 e:000074 BLOCK  C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:715
c:0009 p:0109 s:0072 e:000071 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:966
c:0008 p:0159 s:0065 e:000064 METHOD C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:706
c:0007 p:1910 s:0061 E:000460 TOP    C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/command/fluentd.rb:364 [FINISH]
c:0006 p:---- s:0038 e:000037 CFUNC  :require
c:0005 p:0111 s:0033 e:000032 METHOD C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72
c:0004 p:0096 s:0017 e:000016 TOP    C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/bin/fluentd:15 [FINISH]
c:0003 p:---- s:0013 e:000012 CFUNC  :load
c:0002 p:0112 s:0008 E:001020 EVAL   C:/opt/td-agent/bin/fluentd:23 [FINISH]
c:0001 p:0000 s:0003 E:000df0 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
C:/opt/td-agent/bin/fluentd:23:in `<main>'
C:/opt/td-agent/bin/fluentd:23:in `load'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/bin/fluentd:15:in `<top (required)>'
C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
C:/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/command/fluentd.rb:364:in `<top (required)>'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:706:in `run_worker'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:966:in `main_process'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/supervisor.rb:715:in `block in run_worker'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:147:in `run'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/engine.rb:248:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:199:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175:in `lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:175:in `each'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188:in `block in lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:188:in `each'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/root_agent.rb:200:in `block in start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3-x64-mingw32/lib/fluent/compat/call_super_mixin.rb:42:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:638:in `start'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789:in `init_api_client'
C:/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-google-cloud-0.12.5/lib/fluent/plugin/out_google_cloud.rb:1789:in `new'

-- C level backtrace information -------------------------------------------
C:\Windows\SYSTEM32\ntdll.dll(NtWaitForSingleObject+0x14) [0x00007fff67b8fa74]
C:\Windows\System32\KERNELBASE.dll(WaitForSingleObjectEx+0x93) [0x00007fff63f385c3]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_bugreport+0x37b) [0x00000000653c96ab]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_bug_for_fatal_signal+0x7f) [0x00000000651e432f]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(ruby_sysinit+0x790) [0x0000000065322030]
 [0x0000000000402436]
C:\Windows\System32\msvcrt.dll(_C_specific_handler+0x98) [0x00007fff675e8068]
C:\Windows\SYSTEM32\ntdll.dll(_chkstk+0x11f) [0x00007fff67b94a2f]
C:\Windows\SYSTEM32\ntdll.dll(RtlWalkFrameChain+0x14bf) [0x00007fff67af4cef]
C:\Windows\SYSTEM32\ntdll.dll(KiUserExceptionDispatcher+0x2e) [0x00007fff67b9379e]
 [0xffffffffa1470000]
C:\opt\td-agent\lib\ruby\gems\2.7.0\gems\grpc-1.45.0-x64-mingw32\grpc_c.64.ruby(Z15grpc_iomgr_initv+0x6a) [0x0000000009ac8fda]
C:\opt\td-agent\lib\ruby\gems\2.7.0\gems\grpc-1.45.0-x64-mingw32\grpc_c.64.ruby(grpc_init+0x86) [0x0000000009b54066]
 [0x000000007104835c]
 [0x000000007104664f]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_undefined_alloc+0xc0) [0x0000000065294800]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x23c) [0x00000000653abf2c]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_yield+0x307) [0x00000000653b03b7]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_ary_each+0x3d) [0x0000000065145afd]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x9ee) [0x00000000653ba21e]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x23c) [0x00000000653abf2c]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_yield+0x307) [0x00000000653b03b7]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_ary_each+0x3d) [0x0000000065145afd]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x9ee) [0x00000000653ba21e]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_get_expanded_load_path+0x13f1) [0x0000000065250251]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_f_require+0x22) [0x0000000065250412]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_resolve_feature_path+0x80a) [0x000000006524ecda]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x338) [0x00000000653a1bb8]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_invoke_bmethod+0x131b) [0x00000000653adb0b]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_check_funcall_default+0x8f6) [0x00000000653ba126]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x9e5) [0x00000000653ac6d5]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(rb_call_end_proc+0x152) [0x00000000651e88f2]
C:\opt\td-agent\bin\x64-msvcrt-ruby270.dll(ruby_run_node+0x95) [0x00000000651ed815]
 [0x0000000000402d49]
 [0x00000000004013b4]
 [0x000000000040150b]
C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x14) [0x00007fff65c87974]

-- Other runtime information -----------------------------------------------

[jagdish-176]

Can anyone help into this, currently we have downgraded to lower version.

[igorpeshansky]

Between 0.12.4 and 0.12.5, the grpc gem was upgraded from 1.31.1 to 1.45.0, which likely introduced this instability. Since we regularly test on Windows with use_grpc enabled, there must be some other factor in your setup that's triggering this. Any information you can provide that would help us reproduce the issue would be helpful.

Line 1789 of out_google_cloud.rb creates a new GRPC::Core::ChannelCredentials object. However, the C stack trace indicates that it crashes in grpc_init (more specifically, in grpc_iomgr_init()), so the credentials thing may be a red herring.

Nonetheless, what kind of credentials are you using to authorize the plugin? What version of Windows do you observe this on? In what environment/platform? Do you see this on Linux as well with the same credentials? Also, from your paths, it looks like you're adding this plugin into an upstream td-agent installation. This plugin is not really supported outside of the Google Cloud logging agent package — do you observe the same crash with that package?