Cyg32 on Win64 fails with stack-protector

Discussion:

Ben RUBSON

2018-04-27 17:12:11 UTC

Hi,

We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.

Issue was found running EncFS FUSE FS using WinFsp.

When forking, and calling StartServiceCtrlDispatcher, program fails with
0xC0000028 / STATUS_BAD_STACK.

Testing several Cygwin versions reveals that issue was introduced between
2017-11-14 and 2017-12-01.
And Bill (WinFsp author) found that compiling with -fno-stack-protector
works around the issue.

There you can then find the full story and Bill's nice investigation :
https://github.com/billziss-gh/winfsp/issues/161

Could it be possible something went wrong with Cygwin ?

Thank you very much for your support !

Best regards,

Ben

Bill Zissimopoulos

2018-04-27 20:29:11 UTC

Permalink

Post by Ben RUBSON
We are facing an issue with Cygwin 2.10 32 bits running on Windows 64
bits.
Issue was found running EncFS FUSE FS using WinFsp.
When forking, and calling StartServiceCtrlDispatcher, program fails with
0xC0000028 / STATUS_BAD_STACK.Testing several Cygwin versions reveals
that issue
was introduced between 2017-11-14 and 2017-12-01. And Bill (WinFsp
author) found > that compiling with -fno-stack-protector works around
the issue.There you can
https://github.com/billziss-gh/winfsp/issues/161
Could it be possible something went wrong with Cygwin ?

I wanted to add a few more details to Ben’s post, which may help us
further understand and resolve this issue. Ben has done most of the
troubleshooting work on this and I was just lucky enough to capture an
illuminating message in the debugger.

The components that participate in the discussed interaction are:

- EncFS: a POSIX/Cygwin program that implements a FUSE file system. This
program may be built with or without fstack-protector.

- Cygfuse: a Cygwin DLL that provides a FUSE binary interface (i.e. the
fuse_* symbols). This library was built with cygport under Cygwin 2.9
32-bit. I am unclear on whether this means that the library was built with
fstack-protector (likely not).

- Winfsp-x86.dll: a native Windows DLL (32-bit). Built with Visual Studio.

- WOW64: the Windows on Windows environment.

- Win64: the Windows 64-bit OS.

There are no kernel mode components (other than Windows' own) involved in
the discussed interaction.

The interactions are:

- EncFS calls fuse_main_real which resides in libcygfuse.

- Cygfuse loads winfsp-x86.dll and forwards the fuse_main_real call to it.

- Winfsp-x86.dll proceeds to bootstrap the file system. If the command
line does not include the -f switch, then daemon(3) is called. This is
done by calling back into the Cygfuse Cygwin DLL.

- Cygfuse calls daemon(3). On return it reloads/reinits winfsp-x86.dll,
because Cygwin's fork does not handle native DLL’s. It then returns
control to (the possibly reloaded!) winfsp-x86.dll.

- Winfsp-x86.dll proceeds to initialize the FUSE file system. It calls
StartServiceCtrlDispatcher in an attempt to convert the process into a
service (I wanted FUSE file systems on Windows to be easily converted into
Windows services. Failure of the StartServiceCtrlDispatcher is benign and
the WinFsp-FUSE infrastructure knows how to handle it.)

Now if EncFS is compiled under Cygwin 2.9 (32-bit or 64-bit) or it is
compiled with fno-stack-protector under Cygwin 2.10 everything works just
fine. If however EncFS is compiled with the default (presumably it
includes fstack-protector) then the EncFS daemonized process dies without
any other information.

Attaching the debugger to the daemonized process reveals that while
unwinding an (likely benign) exception deep inside
StartServiceCtrlDispatcher, RtlUnwindEx (or its friends) determines that
the stack looks wrong and raises STATUS_BAD_STACK, which is rather fatal.

Corinna Vinschen

2018-05-29 16:59:58 UTC

Permalink

Hi Ben,

Post by Ben RUBSON
Hi,
We are facing an issue with Cygwin 2.10 32 bits running on Windows 64 bits.
Issue was found running EncFS FUSE FS using WinFsp.
When forking, and calling StartServiceCtrlDispatcher, program fails with
0xC0000028 / STATUS_BAD_STACK.
Testing several Cygwin versions reveals that issue was introduced between
2017-11-14 and 2017-12-01.
And Bill (WinFsp author) found that compiling with -fno-stack-protector
works around the issue.
https://github.com/billziss-gh/winfsp/issues/161
Could it be possible something went wrong with Cygwin ?
Thank you very much for your support !

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.

I suggest to build your own Cygwin and bisect it. Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method. Make sure the cygwin1.dbg
file can be found by GDB.

HTH,
Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Bill Zissimopoulos

2018-05-30 19:29:18 UTC

Permalink

Corinna, thanks for your response.

There appear to be some SSP related commits in the discussed time frame.
For example:

https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
663a54fa1a7de1a6c4a0a3316a6e

Another possibility would be a change in the gcc default options between
Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
2.10 compiles with it enabled by default?

Bill

Post by Corinna Vinschen
Hi Ben,

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.
I suggest to build your own Cygwin and bisect it. Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method. Make sure the cygwin1.dbg
file can be found by GDB.
HTH,
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Corinna Vinschen

2018-05-31 08:18:35 UTC

Permalink

Post by Bill Zissimopoulos
Corinna, thanks for your response.

Post by Corinna Vinschen
Hi Ben,

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.
I suggest to build your own Cygwin and bisect it. Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method. Make sure the cygwin1.dbg
file can be found by GDB.

There appear to be some SSP related commits in the discussed time frame.
https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
663a54fa1a7de1a6c4a0a3316a6e
Another possibility would be a change in the gcc default options between
Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
2.10 compiles with it enabled by default?
Bill

I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
my own Cygwin DLL centric view.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Ken Brown

2018-05-31 13:13:01 UTC

Permalink

Post by Corinna Vinschen

Post by Bill Zissimopoulos
Corinna, thanks for your response.

Post by Corinna Vinschen
Hi Ben,

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.
I suggest to build your own Cygwin and bisect it. Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method. Make sure the cygwin1.dbg
file can be found by GDB.

I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
my own Cygwin DLL centric view.

I think you meant to CC a different Ken.

Ken

Corinna Vinschen

2018-06-01 10:06:44 UTC

Permalink

Post by Ken Brown

Post by Corinna Vinschen

Post by Bill Zissimopoulos
Corinna, thanks for your response.

Post by Corinna Vinschen
Hi Ben,

Even with Bill's additionl info I have no idea what change in the
above time frame might be the culprit, especially since no change
has been made to stack handling.
I suggest to build your own Cygwin and bisect it. Also, you
may want to run your stuff under GDB and set breakpoints to, e.g.,
Cygwin's exception::handle method. Make sure the cygwin1.dbg
file can be found by GDB.

I CCed Ken (gcc) and Yaakov (SSP) to get more qualified input than
my own Cygwin DLL centric view.

I think you meant to CC a different Ken.

Oops, I meant to CC a JonY, in fact, not a Ken at all. Sorry!

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

JonY

2018-06-01 10:49:47 UTC

Permalink

Post by Bill Zissimopoulos
There appear to be some SSP related commits in the discussed time frame.
https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0a5dfdbd1ba3
663a54fa1a7de1a6c4a0a3316a6e
Another possibility would be a change in the gcc default options between
Cygwin 2.9 and Cygwin 2.10. I do not know if this is possible, but perhaps
gcc in 2.9 compiles with -fstack-protector disabled by default and gcc in
2.10 compiles with it enabled by default?

libssp in gcc was disabled in favor of newlib's, not sure if that has
something to do with it too.

Ben RUBSON

2018-06-20 16:42:32 UTC

Permalink

Bill, Corinna, Jony, thank you very much for your answers and details on
this topic.
And sorry for my late answer, I was totally stuck on some critical subjects
last days...

Post by JonY

libssp in gcc was disabled in favor of newlib's, not sure if that has
something to do with it too.

Jony, do you mean that compilation options have changed, which could lead
to our issue ?
Would you manage to find out which change provided this ?
By any chance, between 2017-11-14 and 2017-12-01 ?

Thank you again,

Best regards,

Ben

JonY

2018-06-21 09:42:49 UTC

Permalink

Post by Ben RUBSON

Post by JonY
libssp in gcc was disabled in favor of newlib's, not sure if that has
something to do with it too.

Jony, do you mean that compilation options have changed, which could
lead to our issue ?
Would you manage to find out which change provided this ?
By any chance, between 2017-11-14 and 2017-12-01 ?

The SSP implementation has changed, may be related to
https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=6a848db44242a24d6570ef7994918c66ce483fd2

Ben RUBSON

2018-06-20 16:50:42 UTC

Permalink

Post by Corinna Vinschen
Hi Ben,

Corinna, could you elaborate please ?
I'm not sure to clearly understand what you mean by bisecting Cygwin.

Thank you !

Ben

Ford, Brian

2018-06-20 17:21:21 UTC

Permalink

She means a binary search of source revisions to isolate when the problem was first introduced.
--
Brian Ford - Systems Architect
FlightSafety Visual Systems

Post by Ben RUBSON

Post by Corinna Vinschen
Hi Ben,

Corinna, could you elaborate please ?
I'm not sure to clearly understand what you mean by bisecting Cygwin.
Thank you !
Ben

Corinna Vinschen

2018-06-21 07:21:05 UTC

Permalink

Post by Ford, Brian

Post by Ben RUBSON

Post by Corinna Vinschen
Hi Ben,

Corinna, could you elaborate please ?
I'm not sure to clearly understand what you mean by bisecting Cygwin.
Thank you !
Ben

She means a binary search of source revisions to isolate when the problem was first introduced.

Right, `git bisect'.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Ben RUBSON

2018-06-23 13:50:40 UTC

Permalink

Post by Corinna Vinschen
I suggest to build your own Cygwin and bisect it.

So I tried to bisect and had some difficulties.
I used a fresh 32 bits installation on a 64 bits system and followed this
howto :
https://cygwin.com/faq/faq.html#faq.programming.building-cygwin

Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions
(which is the one JonY suspects)

With gcc 7.3, compilation fails with undefined __stack_chk_fail,
__stack_chk_guard etc...
With gcc 6.4, compilation succeeds. Cygwin correctly starts with the new
cygwin1.dll, but then, some programs, such as cmake (cmake --help) do not
show any output anymore.
Same behavior using Cygwin snapshots. As it worked a few weeks ago, I tried
to downgrade some runtime libraries, but with no luck.
This is easily reproductible, just install Cygwin32 on a 64-bits system,
and select Cygwin 2.9 library instead of the 2.10 one.

So, unfortunately, I was not able to test any commit prior to the one given
above.
This commit however correctly compiles with both gcc 6.4 and 7.3.
Cygwin correctly starts with it, program correctly show their output. And
I'm able to reproduce the issue I opened this topic for.
So something important has changed between 2017-11-14 and 2017-11-29 which
triggers this bug.

Perhaps the new SSP mechanism is more powerful than the previous one and
triggers a long-time existing bug ?

Thank you again,

Ben

Corinna Vinschen

2018-06-25 09:44:48 UTC

Permalink

Yaakov, can you take a look, please?

Thanks,
Corinna

Post by Ben RUBSON

Post by Corinna Vinschen
I suggest to build your own Cygwin and bisect it.

So I tried to bisect and had some difficulties.
I used a fresh 32 bits installation on a 64 bits system and followed this
https://cygwin.com/faq/faq.html#faq.programming.building-cygwin
Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions
(which is the one JonY suspects)
With gcc 7.3, compilation fails with undefined __stack_chk_fail,
__stack_chk_guard etc...
With gcc 6.4, compilation succeeds. Cygwin correctly starts with the new
cygwin1.dll, but then, some programs, such as cmake (cmake --help) do not
show any output anymore.
Same behavior using Cygwin snapshots. As it worked a few weeks ago, I tried
to downgrade some runtime libraries, but with no luck.
This is easily reproductible, just install Cygwin32 on a 64-bits system, and
select Cygwin 2.9 library instead of the 2.10 one.
So, unfortunately, I was not able to test any commit prior to the one given
above.
This commit however correctly compiles with both gcc 6.4 and 7.3.
Cygwin correctly starts with it, program correctly show their output. And
I'm able to reproduce the issue I opened this topic for.
So something important has changed between 2017-11-14 and 2017-11-29 which
triggers this bug.
Perhaps the new SSP mechanism is more powerful than the previous one and
triggers a long-time existing bug ?
Thank you again,
Ben

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Carter, Mark Andrew (Andy)

2018-06-25 15:35:47 UTC

Permalink

We use Cyg32 on a Win64 system and have encountered a gcc compiler bug causing stack corruption due to an invalid optimized stack allocation for a function call that return a large structure (40 bytes). Possibly related to the long standing bug reported here "https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83116". This bug could bite with compiler version / switch settings or otherwise valid code changes.

Andy Carter - Senior Staff Software Engineer
FVS - FVS Engineering - 5695 Campus Parkway - Hazelwood, MO, 63042 - United States
Tel: 314.551.8573
***@FlightSafety.com - flightsafety.com - A Berkshire Hathaway company

-----Original Message-----
From: cygwin-developers-***@cygwin.com [mailto:cygwin-developers-***@cygwin.com] On Behalf Of Corinna Vinschen
Sent: Monday, June 25, 2018 4:45 AM
To: cygwin-***@cygwin.com
Cc: Yaakov Selkowitz <***@redhat.com>
Subject: Re: Cyg32 on Win64 fails with stack-protector

Yaakov, can you take a look, please?

Thanks,
Corinna

Post by Ben RUBSON

Post by Corinna Vinschen
I suggest to build your own Cygwin and bisect it.

So I tried to bisect and had some difficulties.
I used a fresh 32 bits installation on a 64 bits system and followed
https://cygwin.com/faq/faq.html#faq.programming.building-cygwin
Prior to this commit : 6a848db44242a24d6570ef7994918c66ce483fd2
2017-11-29 - Yaakov Selkowitz - cygwin: export SSP functions (which is
the one JonY suspects)
With gcc 7.3, compilation fails with undefined __stack_chk_fail,
__stack_chk_guard etc...
With gcc 6.4, compilation succeeds. Cygwin correctly starts with the
new cygwin1.dll, but then, some programs, such as cmake (cmake --help)
do not show any output anymore.
Same behavior using Cygwin snapshots. As it worked a few weeks ago, I
tried to downgrade some runtime libraries, but with no luck.
This is easily reproductible, just install Cygwin32 on a 64-bits
system, and select Cygwin 2.9 library instead of the 2.10 one.
So, unfortunately, I was not able to test any commit prior to the one
given above.
This commit however correctly compiles with both gcc 6.4 and 7.3.
Cygwin correctly starts with it, program correctly show their output.
And I'm able to reproduce the issue I opened this topic for.
So something important has changed between 2017-11-14 and 2017-11-29
which triggers this bug.
Perhaps the new SSP mechanism is more powerful than the previous one
and triggers a long-time existing bug ?
Thank you again,
Ben

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer

Ben RUBSON

2018-06-25 16:08:46 UTC

Permalink

Post by Carter, Mark Andrew (Andy)
We use Cyg32 on a Win64 system and have encountered a gcc compiler bug
causing stack corruption due to an invalid optimized stack allocation for
a function call that return a large structure (40 bytes). Possibly
related to the long standing bug reported here
"https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83116". This bug could
bite with compiler version / switch settings or otherwise valid code
changes.

Thank you Andy for your help.
About the bug you point to, sounds like it has been corrected in GCC 7.3 :
https://gcc.gnu.org/viewcvs/gcc/tags/gcc_7_3_0_release/gcc/cp/ChangeLog?view=markup#l30
Unfortunately I also face the issue with this GCC version.

Ben