MSYS mode (continue)

struct utsname
{
- char sysname[20];
+ char sysname[21];

I can't drop any symbol from "MINGW32" word :)

That's not 20 chars long.

CYGWIN_NT-6.1-WOW64\0
But
MINGW32_NT-6.1-WOW64\0
is 21 symbol.

Then drop the dash between "NT" and "6.1". Or let's drop the WOW64
since it's redundant anyway, given the content of "machine".

Let's drop it!

Can we back up a moment and discuss whether, regardless of string
length, this is the right approach? This isn't what I'd envisioned
when I said "hooks".

I don't think hooks make sense for such simple, nonintrusive stuff.
This may be different for bigger things like the weird "copy symlinks"
stuff, of course.
Also, you didn't so far define how these hooks are supposed to work.
A detailed description of your idea would be useful for the discussion.

http://cygwin.com/ml/cygwin/2013-06/msg00515.html

I thought it was clear that this would be a general mechanism which
could be used to modify Cygwin's behavior.

So, in this scenario, you would:

callout (CO_UNAME, &name);

We would provide a cygwin_internal method for setting up the callout
hook. If that wasn't set up then obviously callout would be a no-op.

I don't think that it makes sense for there to be two mechanisms to
accomplish the goal of allowing another DLL to modify Cygwin's behavior.

Obviously a MSYS helper DLL would have to do the setup early, maybe
in its DLL initialization.

Corinna Vinschen

2013-07-11 11:17:44 UTC

Post by Corinna Vinschen
I don't think hooks make sense for such simple, nonintrusive stuff.
This may be different for bigger things like the weird "copy symlinks"
stuff, of course.
Also, you didn't so far define how these hooks are supposed to work.
A detailed description of your idea would be useful for the discussion.

http://cygwin.com/ml/cygwin/2013-06/msg00515.html
I thought it was clear that this would be a general mechanism which
could be used to modify Cygwin's behavior.
callout (CO_UNAME, &name);
We would provide a cygwin_internal method for setting up the callout
hook. If that wasn't set up then obviously callout would be a no-op.
I don't think that it makes sense for there to be two mechanisms to
accomplish the goal of allowing another DLL to modify Cygwin's behavior.
Obviously a MSYS helper DLL would have to do the setup early, maybe
in its DLL initialization.

The hooking itself is the lesser problem. What I'm rather wondering
about is how the MSYS helper DLL comes into action.

Ideally, MSYS executables would actually be Cygwin executables, linked
against the Cygwin DLL. The advantage being that the executable could
run in an MSYS and a Cygwin environment, whatever it happens to be
dropped into.

The MSYS DLL could be loaded from an MSYS specific crt0.o, which tries
to load the MSYS DLL dynamically. Either loading the MSYS DLL works or
it doesn't, but if it works, it could just call cygwin_internal in it's
dll_entry function to set up the hooking mechanism. If it fails, the
executable works as a plain Cygwin executable.

Alternatively, MSYS crt0.o itself provides all the necessary functionality,
which might be a lot easier to implement. It could call cygwin_internal
and provide the necessary callbacks and it would be linked against the
Cygwin DLL "just so".

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Alexey Pavlov

2013-07-25 11:06:11 UTC

Is this thread is interesting yet or we can leave all as is?

Regards,
Alexey.

Corinna Vinschen

2013-07-25 11:11:06 UTC

Post by Alexey Pavlov
Is this thread is interesting yet or we can leave all as is?

This is of course interesting. That's why I sent my mail on the 11th.
A reply to the points raised and further discussion would help a lot.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Charles Wilson

2013-07-25 13:11:07 UTC

This post might be inappropriate. Click to display it.

Corinna Vinschen

2013-07-25 15:02:09 UTC

Post by Corinna Vinschen
The hooking itself is the lesser problem. What I'm rather wondering
about is how the MSYS helper DLL comes into action.
Ideally, MSYS executables would actually be Cygwin executables, linked
against the Cygwin DLL. The advantage being that the executable could
run in an MSYS and a Cygwin environment, whatever it happens to be
dropped into.
The MSYS DLL could be loaded from an MSYS specific crt0.o, which tries
to load the MSYS DLL dynamically. Either loading the MSYS DLL works or
it doesn't, but if it works, it could just call cygwin_internal in it's
dll_entry function to set up the hooking mechanism. If it fails, the
executable works as a plain Cygwin executable.

I dunno. Back in 2009, I (re)built and updated to then-current EVERY
single element of MinGW.org's msys distribution. There were a LOT
of patches required, which were not required on cygwin.
[...]

In how far does this affect us today? The idea to have a Cygwin DLL
underneath and just add the MSYS functionality by linking one way or
the other allows to get rid of lots of old workarounds. There's no
reason to stick with the old ways.

Post by Charles Wilson
However, many of the patches were to handle differences inherent in
the environment. For example, msys-make kept the "allow X:/foo/bar
paths in rules" patch, which cgf long ago removed from cygwin-make.
[...]

This isn't relevant. The idea is to have a different crt0.o which
allows to get the MSYS specific functionaily by building an application.
against it. This does not reduce the necessity to patch applications
which require special functionality like a DOS-path-resistant make.

Post by Charles Wilson
But some apps (and our config-foo scripts!) check those settings;
for instance, ssh.exe wants to ensure that ~/.ssh is not writable by
anybody except the owner. That stuff had to be patched out, to work
on msys.

Which is bad. I removed some older code from ssh specificially per the
request from the upstream devs since they thought it's too unsecure.
It's kind of frustrating having others adding this stuff back in just
to stay happily unsecure. Windows 95 is long gone.

Post by Charles Wilson
Well, we don't want $real-cygwin's ssh.exe to act in such an
insecure manner, but then cygwin-as-msys users lose out, unless then
only use the ssh.exe from their parallel installation of
$real-cygwin.

And the disadvantage is...? The whole idea is to be able to stick to
full-fledged Cygwin functionalitiy, except for the cases where it
revolves around building native applications and DOS path handling.
There's no good reason to duplicate everything in an incompatible
way.

Post by Corinna Vinschen
Alternatively, MSYS crt0.o itself provides all the necessary functionality,
which might be a lot easier to implement. It could call cygwin_internal
and provide the necessary callbacks and it would be linked against the
Cygwin DLL "just so".

I think you would actually need an entirely separate distribution of
tools,

That was the point, Chuck. The idea was to have an MSYS distro with
tools linked against the special crt0.o to get the aforementioned extra
functionality. But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality. You could have your cake *and* eat it.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Charles Wilson

2013-07-25 18:20:50 UTC

I dunno. Back in 2009, I (re)built and updated to then-current EVERY
single element of MinGW.org's msys distribution. There were a LOT
of patches required, which were not required on cygwin.
[...]

Right, that's what the paragraph you snipped was about.

Ah, but you're assuming that the app itself is *recompiled* so that it
links a different crt0.o. I interpreted your statement:
"The advantage being that the executable could run in
an MSYS and a Cygwin environment, whatever it happens
to be dropped into."
to mean "the exact same binary". That misunderstanding is what led to
most of the confusion below.

My point is, I don't think *that* would work, for all the reasons I
detailed.

Of course it's bad. MSYS(current version), based on cygwin-1.3.x AND
with an even more broken security model than 1.3.x had, sux when
compared to modern day cygwin.

However...a LOT of people don't care about file security when operating
in a network environment that is deemed "protected" from outside
interference. E.g. network shares on an internal lab network, etc. At
$dayjob, I've seen folks get really annoyed by the issues that crop up
when trying to mix native tools and cygwin's security model [1].

[1] the magic
----------+ 1 me mygroup 1.4M Jul 25 11:51 some-file
permssion settings you get when some-file was created using win32 tools.

So, they prefer MSYS no-security model, because in a protected network,
they just want to get their job done, and not spend time chmod'ing stuff
(or, in severe cases you can get yourself into a jam where you don't
even have permission to "fix" it, and you have to get some IT support
guy with Local Administrator privilege to do it. "And how did you create
this file? What where you doing?").

So...if your msys-on-cygwin implements $dumb-security, then...no ssh at
all? (Geez, I've seen people store their ~/.ssh on a network share,
which is...counterindicated, at best. But it's still done.)

I thought the "whole idea" was to replicate the existing msys toolkit
(which currently includes rsh, ssh, and an extremely limited subset of
the cygwin universe), only with a better, more maintainable core msys
dll (which is actually cygwin1.dll + the extra hook-impl-providing msys
thingy).

I think you would actually need an entirely separate distribution of
tools,

That was the point, Chuck. The idea was to have an MSYS distro with
tools linked against the special crt0.o to get the aforementioned extra
functionality.

Ah, so you're not talking about dropping in "the same executable" into
the MSYS arena and expecting its behavior to magically change.
C:\cygwin-msys\bin\make.exe is a different binary than
C:\cygwin\bin\make.exe.

OK so far.

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.

But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?

Naturally from the command line I can compensate:

msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )

but yee gods that'd be annoying in any automated setting.

--
Chuck

Charles Wilson

2013-07-25 18:33:02 UTC

Post by Corinna Vinschen
This isn't relevant. The idea is to have a different crt0.o which
allows to get the MSYS specific functionaily by building an application.
against it. This does not reduce the necessity to patch applications
which require special functionality like a DOS-path-resistant make.

Ah, but you're assuming that the app itself is *recompiled* so that it
"The advantage being that the executable could run in
an MSYS and a Cygwin environment, whatever it happens
to be dropped into."
to mean "the exact same binary". That misunderstanding is what led to
most of the confusion below.
My point is, I don't think *that* would work, for all the reasons I
detailed.

Wait, ignore the rest of the previous message. I think I get where
you're going now, but I'll need to think on it some more.

Going to be AFK for a day or so, tho...

--
Chuck

Christopher Faylor

2013-07-25 20:53:20 UTC

This post might be inappropriate. Click to display it.

LRN

2013-07-25 21:07:59 UTC

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

I don't know if this helps but the vague plan is to now have two DLLs
where before you only had one. You'd still be providing "MSYS" binaries
which relied on "MSYS.dll" but, under the hood, MSYS.dll would be only a
small dll which relied on cygwin1.dll for all of the heavy lifting.
You'd still have a normal MSYS distribution and it would still, in theory,
support everything (with the possible exception of very lax security) that
the old MSYS did. An MSYS release would consist of MSYS*.dll, cygwin1.dll,
bash, etc.

Out of curiosity: why do you insist on having MSYS functionality in a
separate dll, when it could be just part of cygwin1.dll (disableable and
enableable in the same way other Cygwin features are disabled/enabled -
via CYGWIN envvar)? What advantage would that give, that justifies the
increase in implementation complexity (hooking up the dll, etc)? Was
that justified earlier in this thread and i just neglected to read that
far back?

Also, i'm planning on building experimental cygwin dll that doesn't use
NO_ACL on mounts by default (msys2 always uses NO_ACL on mounts by
default) and see if it works for me. If it does, that'll remove yet
another item from the list of msys vs cygwin differences, at least for me.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Larry Hall (Cygwin Developers)

2013-07-25 21:30:57 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Yes, it was mentioned. The idea is to keep the MSYS-specific stuff separate
so Cygwin functionality isn't impacted by what MSYS wants. This reduces
the complexity of the Cygwin code at least.

--
Larry

Christopher Faylor

2013-07-26 01:55:17 UTC

Post by Larry Hall (Cygwin Developers)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Yes, it was mentioned. The idea is to keep the MSYS-specific stuff separate
so Cygwin functionality isn't impacted by what MSYS wants. This reduces
the complexity of the Cygwin code at least.

Right. Thanks Larry.

The bottom line is that we don't want there to be a bunch of "if
(is_msys)"'s in the code. We don't want to have to worry about how msys
does things. I don't want to have to inspect msys patches, except for
requests for a new hook.

A hook will be a fast no-op in the pure cygwin cases. There would be
very little "implementation complexity".

cgf

LRN

2013-07-26 04:02:45 UTC

Post by Larry Hall (Cygwin Developers)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Yes, it was mentioned. The idea is to keep the MSYS-specific stuff separate
so Cygwin functionality isn't impacted by what MSYS wants. This reduces
the complexity of the Cygwin code at least.

Right. Thanks Larry.
The bottom line is that we don't want there to be a bunch of "if
(is_msys)"'s in the code. We don't want to have to worry about how msys
does things. I don't want to have to inspect msys patches, except for
requests for a new hook.
A hook will be a fast no-op in the pure cygwin cases. There would be
very little "implementation complexity".

OK.

I've attached the patch that alexey gave me recently, if pointless
cygwin->msys renames are removed from msys2, this is what msys2 boils
down to. Please specify how the hooks will look, now that you know which
points of the code need to be hooked up.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Christopher Faylor

2013-07-26 05:46:15 UTC

Please specify how the hooks will look, now that you know which points
of the code need to be hooked up.

Apparently you haven't read any of the past discussion and are just
trying to jump into the middle without doing any homework.

We are still discussing implementation. That is the whole point of
this thread.

I did provide a straw man proposal which showed what a hook could look
like, however.

cgf

Corinna Vinschen

2013-07-26 08:15:10 UTC

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Here's where I disagree. I think the executables should *not* rely on
MSYS.dll being available. Ideally the executables are linked against
the Cygwin DLL and MSYS.dll is called as a side-by-side implementation.
So, if MSYS.dll isn't available, they still function as normal Cygwin
executables. That's why I proposed the solution(s) in
http://cygwin.com/ml/cygwin-developers/2013-07/msg00003.html

Assuming you implement it the first way, you get an executable linked
against a crt0.o which tries a LoadLibrary("MSYS.dll"). If it fails,
the executable does business as usual. It will not fail, because
there's still the Cygwin DLL.

If LoadLib worked, crt0 calls GetProcAddress("__msys_hook") and then
__msys_hook(). __msys_hook collects the hook function pointers and
sends them to the CYgwin DLL via a call to cygwin_internal(CW_HOOK,
&hook_list). Voila, the hooks are set up, we're in MSYS mode.

Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Post by Christopher Faylor
I don't think anyone was proposing seamless interoperation between MSYS
and cygwin.

Yes, I honestly think this would be possible and desirable. MSYS is
just a tiny change for a specific task in comparison to the default
Cygwin mode. MSYS would concentrate on this task and the required tools
for this task and the rest could be stock Cygwin distro.

Btw., this does *not* mean I agree with all changes MSYS is doing. I
have a hard time to see the necessity changing the /etc/fstab layout,
for instance, since it doesn't add or change anything you can't have
with the standard fstab.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Christopher Faylor

2013-07-26 15:14:32 UTC

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Here's where I disagree. I think the executables should *not* rely on
MSYS.dll being available. Ideally the executables are linked against
the Cygwin DLL and MSYS.dll is called as a side-by-side implementation.
So, if MSYS.dll isn't available, they still function as normal Cygwin
executables. That's why I proposed the solution(s) in
http://cygwin.com/ml/cygwin-developers/2013-07/msg00003.html
Assuming you implement it the first way, you get an executable linked
against a crt0.o which tries a LoadLibrary("MSYS.dll"). If it fails,
the executable does business as usual. It will not fail, because
there's still the Cygwin DLL.
If LoadLib worked, crt0 calls GetProcAddress("__msys_hook") and then
__msys_hook(). __msys_hook collects the hook function pointers and
sends them to the CYgwin DLL via a call to cygwin_internal(CW_HOOK,
&hook_list). Voila, the hooks are set up, we're in MSYS mode.
Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Post by Christopher Faylor
I don't think anyone was proposing seamless interoperation between MSYS
and cygwin.

Yes, I honestly think this would be possible and desirable. MSYS is
just a tiny change for a specific task in comparison to the default
Cygwin mode. MSYS would concentrate on this task and the required tools
for this task and the rest could be stock Cygwin distro.
Btw., this does *not* mean I agree with all changes MSYS is doing. I
have a hard time to see the necessity changing the /etc/fstab layout,
for instance, since it doesn't add or change anything you can't have
with the standard fstab.

So, you did. I missed the ramifications of that message. I guess
that's why Chuck was confused. I'm suggesting something different so
we're all confused.

This does not address the problem that there are local modifications in
some MSYS utilities. This doesn't solve the problem of make
understanding c:\foo. Doesn't MSYS bash allow constructs of the form
c:\foo without making that c:foo? If so, the DLL isn't going to help
there.

I don't think that we should be seeing the word "msys" throughout the
DLL source code. The hooks could all be just named "cygwin_hook"
generic and, maybe we should recognize something like a
"CYGWIN=PRELOAD=MSYS.dll". I don't know how the MSYS folks would
feel about that however. Having to tell their users that they need
to set an environment variable before they do anything seems like it
would be a maintenance headache.

As far as /etc/fstab is concerned, since they were the first to
implement the notion, I would assume that they don't want to have to
tell their userbase to change any more than we would want to have to
tell every Cygwin user that they have to edit a file to make the
next version of Cygwin work correctly.

cgf

LRN

2013-07-26 15:48:13 UTC

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Here's where I disagree. I think the executables should *not* rely on
MSYS.dll being available. Ideally the executables are linked against
the Cygwin DLL and MSYS.dll is called as a side-by-side implementation.
So, if MSYS.dll isn't available, they still function as normal Cygwin
executables. That's why I proposed the solution(s) in
http://cygwin.com/ml/cygwin-developers/2013-07/msg00003.html
Assuming you implement it the first way, you get an executable linked
against a crt0.o which tries a LoadLibrary("MSYS.dll"). If it fails,
the executable does business as usual. It will not fail, because
there's still the Cygwin DLL.
If LoadLib worked, crt0 calls GetProcAddress("__msys_hook") and then
__msys_hook(). __msys_hook collects the hook function pointers and
sends them to the CYgwin DLL via a call to cygwin_internal(CW_HOOK,
&hook_list). Voila, the hooks are set up, we're in MSYS mode.
Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Post by Christopher Faylor
I don't think anyone was proposing seamless interoperation between MSYS
and cygwin.

Yes, I honestly think this would be possible and desirable. MSYS is
just a tiny change for a specific task in comparison to the default
Cygwin mode. MSYS would concentrate on this task and the required tools
for this task and the rest could be stock Cygwin distro.
Btw., this does *not* mean I agree with all changes MSYS is doing. I
have a hard time to see the necessity changing the /etc/fstab layout,
for instance, since it doesn't add or change anything you can't have
with the standard fstab.

Yep, you need make sure that make is not configured with ac_cv_dos_paths=no.

Post by Christopher Faylor
Doesn't MSYS bash allow constructs of the form
c:\foo without making that c:foo?

It doesn't. You have to carefully avoid the use of '\' everywhere (much
easier than escaping it).

Post by Christopher Faylor
I don't think that we should be seeing the word "msys" throughout the
DLL source code. The hooks could all be just named "cygwin_hook"
generic and, maybe we should recognize something like a
"CYGWIN=PRELOAD=MSYS.dll". I don't know how the MSYS folks would
feel about that however. Having to tell their users that they need
to set an environment variable before they do anything seems like it
would be a maintenance headache.

MSYS actually does require an environment variable to be set.
MSYSTEM=MINGW32 or MSYSTEM=MSYS (at least MSYS1 did, if you wanted to
use it with MinGW, and MSYS2 does, in its current form). Most users must
have MSYSTEM=MINGW32, otherwise their build system type will be
misdetected (as MSYSTEM affects the name returned by uname() syscall).

In MSYS1 this was done by the msys.bat batch file that was used to
launch the shell.

Post by Christopher Faylor
As far as /etc/fstab is concerned, since they were the first to
implement the notion, I would assume that they don't want to have to
tell their userbase to change any more than we would want to have to
tell every Cygwin user that they have to edit a file to make the
next version of Cygwin work correctly.

I wouldn't worry about this too much. MSYS2, being nearly pure Cygwin,
may require more elaborate installation procedures than MSYS1 did. These
procedures could include an /etc/fstab conversion script, if users
upgrade from MSYS1.

MSYS1 compatibility might be interesting for some people (for mingw.org
- - definitely), but not for everyone.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Corinna Vinschen

2013-07-26 16:16:49 UTC

MSYS actually does require an environment variable to be set.
MSYSTEM=MINGW32 or MSYSTEM=MSYS (at least MSYS1 did, if you wanted to
use it with MinGW, and MSYS2 does, in its current form). Most users must
have MSYSTEM=MINGW32, otherwise their build system type will be
misdetected (as MSYSTEM affects the name returned by uname() syscall).
In MSYS1 this was done by the msys.bat batch file that was used to
launch the shell.

So that shouldn't be much of a problem.

I wouldn't worry about this too much. MSYS2, being nearly pure Cygwin,
may require more elaborate installation procedures than MSYS1 did. These
procedures could include an /etc/fstab conversion script, if users
upgrade from MSYS1.
MSYS1 compatibility might be interesting for some people (for mingw.org
- - definitely), but not for everyone.

Wouldn't a generic conversion script be in order then, which also
enforces the noacl setting if desired?

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Corinna Vinschen

2013-07-26 16:12:05 UTC

Post by Corinna Vinschen
But underlying there's still a normal Cygwin DLL and
most tools could just be copied verbatim since they don't need this
extra functionality.

And that's the bit where I disagree. Sure, some scripting tools might
not need adjustment, so long as their interpreter was $MSYS-enabled
(e.g. automake -> msys-perl, msys-bash) -- because the script will "see"
dos-style paths, so its interpreter better be able to handle them.
But unless you restrict yourself to only passing around relative paths
(or god forbid, that old "unity mount" idea), any .exe will need to live
in one world or the other. Otherwise, how would paths be interpreted?
Using which tools' mount table?
msys$ /c/cygwin/bin/foobar.exe $(/c/cygwin/bin/cygpath.exe -u $(cygpath
-d /msys/mount/table/path) )
but yee gods that'd be annoying in any automated setting.

Here's where I disagree. I think the executables should *not* rely on
MSYS.dll being available. Ideally the executables are linked against
the Cygwin DLL and MSYS.dll is called as a side-by-side implementation.
So, if MSYS.dll isn't available, they still function as normal Cygwin
executables. That's why I proposed the solution(s) in
http://cygwin.com/ml/cygwin-developers/2013-07/msg00003.html
Assuming you implement it the first way, you get an executable linked
against a crt0.o which tries a LoadLibrary("MSYS.dll"). If it fails,
the executable does business as usual. It will not fail, because
there's still the Cygwin DLL.
If LoadLib worked, crt0 calls GetProcAddress("__msys_hook") and then
__msys_hook(). __msys_hook collects the hook function pointers and
sends them to the CYgwin DLL via a call to cygwin_internal(CW_HOOK,
&hook_list). Voila, the hooks are set up, we're in MSYS mode.
Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Post by Christopher Faylor
I don't think anyone was proposing seamless interoperation between MSYS
and cygwin.

Yes, I honestly think this would be possible and desirable. MSYS is
just a tiny change for a specific task in comparison to the default
Cygwin mode. MSYS would concentrate on this task and the required tools
for this task and the rest could be stock Cygwin distro.
Btw., this does *not* mean I agree with all changes MSYS is doing. I
have a hard time to see the necessity changing the /etc/fstab layout,
for instance, since it doesn't add or change anything you can't have
with the standard fstab.

That's what I talked about in my next to last paragraph. As far as I'm
concerned, MSYS would be a DLL tweaking the behaviour of Cygwin via the
hooks, *plus* a set of tools specificially tweaked where necessary for
what it's supposed to do. Make is one example, bash another, gcc would
target mingw32 by default, stuff like that. Still, that doesn't mean
that every tool has to be tweaked. Most tools would work OOTB.

Post by Christopher Faylor
I don't think that we should be seeing the word "msys" throughout the
DLL source code. The hooks could all be just named "cygwin_hook"
generic

Actually, I'd prefer to have only a single foreign environment using
the Cygwin DLL, not a whole bunch. This implies that I could live
with a single occurence of the work "MSYS" in the Cygwin code.

Post by Christopher Faylor
and, maybe we should recognize something like a
"CYGWIN=PRELOAD=MSYS.dll". I don't know how the MSYS folks would
feel about that however. Having to tell their users that they need
to set an environment variable before they do anything seems like it
would be a maintenance headache.

I don't know if that's a problem for them, but this reminds me... maybe
we can use the LD_PRELOAD mechanism in hookapi.cc for that?

Backward compatibility? What a weird concept :)

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Christopher Faylor

2013-07-26 16:37:08 UTC

Post by Corinna Vinschen
I don't know if that's a problem for them, but this reminds me... maybe
we can use the LD_PRELOAD mechanism in hookapi.cc for that?

I was thinking that we'd use something like that (re my use of PRELOAD
and your use of CW_HOOK) but, you're right. Someone could potentially
implement MSYS.dll via LD_PRELOAD:

LD_PRELOAD=msys.dll

and then have msys.dll hook the functions that it is interested in using
via CW_HOOK.

cgf

Daniel Colascione

2013-07-26 16:44:59 UTC

Post by Corinna Vinschen
I don't know if that's a problem for them, but this reminds me... maybe
we can use the LD_PRELOAD mechanism in hookapi.cc for that?

I was thinking that we'd use something like that (re my use of PRELOAD
and your use of CW_HOOK) but, you're right. Someone could potentially
LD_PRELOAD=msys.dll
and then have msys.dll hook the functions that it is interested in using
via CW_HOOK.

By the way: can we make LD_PRELOAD hook all modules, not just the main program?
If my LD_PRELOADed module hooks write(2) in foo.exe, which uses bar.dll, then
write(2) from foo.exe will be intercepted, but write(2) from bar.dll won't be
--- only foo.exe's import table is munged.

Corinna Vinschen

2013-07-26 16:47:16 UTC

Post by Corinna Vinschen
I don't know if that's a problem for them, but this reminds me... maybe
we can use the LD_PRELOAD mechanism in hookapi.cc for that?

Two points come to mind.

- Does CW_HOOK still work? I had to change a lot of code in
hook_or_detect_cygwin to handle big executables and 32 vs. 64 bit, so
I have no idea if it still does the righht thing in terms of CW_HOOK.

- Isn't CW_HOOK a bit heavy handed for MSYS? MSYS only replaces partial
functionality, not entire functions. Something like a CW_HOOK2 which
only results in Cygwin calling certain function pointers for minor
tweaks could be the result...

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Christopher Faylor

2013-07-26 17:01:00 UTC

Post by Corinna Vinschen
I don't know if that's a problem for them, but this reminds me... maybe
we can use the LD_PRELOAD mechanism in hookapi.cc for that?

Who knows? That's something that should be fixed, regardless.

Post by Corinna Vinschen
- Isn't CW_HOOK a bit heavy handed for MSYS? MSYS only replaces partial
functionality, not entire functions. Something like a CW_HOOK2 which
only results in Cygwin calling certain function pointers for minor
tweaks could be the result...

MSYS is free to call the old function first if it needs to and then modify
the output. That's basically what any hook would do anyway.

Daniel Colascione's point is a stumbling block though since the
LD_PRELOAD method doesn't currently allow hooking anything other than
the main executable. That means that, say, ncurses wouldn't know about
an MSYS-hooked uname. It is possible to work around that but it would
require the addition of new code to the LD_PRELOAD mechanism, requiring
that we keep a table of hooked functions.

cgf

Daniel Colascione

2013-07-26 17:03:42 UTC

Post by Christopher Faylor
Daniel Colascione's point is a stumbling block though since the
LD_PRELOAD method doesn't currently allow hooking anything other than
the main executable. That means that, say, ncurses wouldn't know about
an MSYS-hooked uname. It is possible to work around that but it would
require the addition of new code to the LD_PRELOAD mechanism, requiring
that we keep a table of hooked functions.

You don't need to maintain your own table. When you hook, you just have to munge
all the import tables of all currently-loaded modules, then munge the export
table of the module you want to hook. Munging the export table will take care of
any modules that load in the future.

Christopher Faylor

2013-07-26 17:36:46 UTC

You don't need to maintain your own table. When you hook, you just
have to munge all the import tables of all currently-loaded modules,
then munge the export table of the module you want to hook. Munging
the export table will take care of any modules that load in the future.

I think it would be faster to keep a list of functions since, otherwise,
on fork, it seems like you'd have to walk the whole Cygwin export list,
looking for functions to re-target.

cgf

Yaakov (Cygwin/X)

2013-07-26 23:12:43 UTC

Post by Corinna Vinschen
- Does CW_HOOK still work? I had to change a lot of code in
hook_or_detect_cygwin to handle big executables and 32 vs. 64 bit, so
I have no idea if it still does the righht thing in terms of CW_HOOK.

datefudge (in Ports) seems to work correctly on x86_64, so I believe so.

Yaakov

Charles Wilson

2013-07-27 03:06:50 UTC

Post by Christopher Faylor
I don't know if this helps but the vague plan is to now have two DLLs
where before you only had one. You'd still be providing "MSYS" binaries
which relied on "MSYS.dll" but, under the hood, MSYS.dll would be only a
small dll which relied on cygwin1.dll for all of the heavy lifting.
You'd still have a normal MSYS distribution and it would still, in theory,
support everything (with the possible exception of very lax security) that
the old MSYS did. An MSYS release would consist of MSYS*.dll, cygwin1.dll,
bash, etc.

So, in this proposal I'd have an "msys" directory structure, with
cygwin1.dll, an /etc/fstab, and lots of plain-old-cygwin executables,
bit-for-bit identical to the executables in my "real" cygwin
installation/directory structure.

On executable launch, ALL such applications would check for the hook DLL
(as directed by an env variable, perhaps, or some other mechanism --
maybe encoded in a known-present file, like.../etc/fstab? [1]) and if
present, load it.

Even the (bit-for-bit-identical) apps in my "real" cygwin installation
would do this -- but because (a) they don't have that env setting or
fstab setting, and (b) even if they did, the special msys-hook dll isn't
present in that tree -- so nothing actually happens in that situation.

Now, back in my MSYS-on-cygwin installation, there are SOME executables
that are actually *different* that the corresponding ones in
real-cygwin-land. Stuff like make, bash, perl, etc -- all may have been
compiled with different options because we (the mingw/msys people) want
them to behave differently, in ways that can't automatically be handled
by the hooked changes in cygwin1.dll's own behavior.

Have I got that all correct?

[1] I think this is better than an environment var, because then my
"regular" cygwin tree and my "msys" cygwin tree would both just work,
without needed extraneous global env vars that might interfere with the
other's operation.

In fact, I might want *different* CYGWIN env var settings for the two
trees, but unless I set them in the global env then StartMenu-launched
apps lose out.

Could we maybe extend the CYGWIN env var idea to files, similar to the
/etc/fstab[.d] structure, which are then *augmented* by $CYGWIN?

e.g. in my msys-on-cygwin tree, I might have

/etc/cygwin_settings
hook=/bin/msys-hook.dll
export
glob:ignorecase
winsymlinks:nativestrict
/etc/cygwin_settings.d/user
error_start:c:\msys\bin\gdb.exe

But in my "real" cygwin tree, I might have

/etc/cygwin_settings
proc_retry:1
/etc/cygwin_settings.d/user
pipe_byte

Post by Corinna Vinschen
Assuming you implement it the first way, you get an executable linked
against a crt0.o which tries a LoadLibrary("MSYS.dll"). If it fails,
the executable does business as usual. It will not fail, because
there's still the Cygwin DLL.
If LoadLib worked, crt0 calls GetProcAddress("__msys_hook") and then
__msys_hook(). __msys_hook collects the hook function pointers and
sends them to the CYgwin DLL via a call to cygwin_internal(CW_HOOK,
&hook_list). Voila, the hooks are set up, we're in MSYS mode.
Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Right -- but *some* executables would need to actually BE different,
aside from the underlying posix library's behavioral changes, to get a
"real" MSYS environment.

Post by Christopher Faylor
I don't think anyone was proposing seamless interoperation between MSYS
and cygwin.

I think that would, in fact, work -- but I think the list of tools that
would need compile-time changes is larger than you would hope. But...IMO
that's ok. MSYS's goal was to be a *minimal* system...NOT an alternate
cygwin. If this proposal (a) makes the msys "fork" a non-fork, AND
easier to maintain, and (b) incidentally means that SOMETIMES you might
be able to "drop in" a stock cygwin .exe and it might just work -- no
guarantees, you break it you bought it, we're just mean -- then it's a
win overall.

The MSYS team would just provide patched and (re)compiled versions of
most of their current set of tools...and if users wanted to "drop in"
(e.g.) git.exe, well they are welcome to try it. No support offered or
guaranteed, and they might just get lucky.

Post by Corinna Vinschen
Btw., this does *not* mean I agree with all changes MSYS is doing. I
have a hard time to see the necessity changing the /etc/fstab layout,
for instance, since it doesn't add or change anything you can't have
with the standard fstab.

As cgf pointed out, if "we" (cygwin) are trying to come up with an easy
upgrade path for existing users of MSYS, then we (mingw/msys) users
would prefer not to have to change our existing fstab format on all of
our installations. Unless you (we, cygwin) automate that somehow...

--
Chuck

NightStrike

2013-07-28 00:17:51 UTC

Post by Corinna Vinschen
Another alternative would be if the Cygwin DLL itself had a switch to
load the MSYS dll (export CYGWIN=MSYS ;)). This would allows MSYS mode
even with completely unchanged executables.

Right -- but *some* executables would need to actually BE different, aside
from the underlying posix library's behavioral changes, to get a "real" MSYS
environment.

Perhaps it would be useful to actually identify which packages have
extenuating needs. Maybe it's just one or two. Maybe it's all but
one or two. I don't think that currently, the problem space is
properly enumerated, but is instead living in the abstract.

Corinna Vinschen

2013-07-29 09:29:58 UTC

Post by NightStrike

Right -- but *some* executables would need to actually BE different, aside
from the underlying posix library's behavioral changes, to get a "real" MSYS
environment.

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
changes in relation to a stock Cygwin distro:

- make w/ DOS path handling
- gcc targeting Mingw rather than Cygwin.
- Cygwin compiler provided as cross tool $arch-pc-cygwin-gcc, etc.(*)

Corinna

(*) Per my suggestion there would be no MSYS targeting compiler anymore.
It would target Cygwin and everything else just depends on how the
MSYS dll gets loaded.

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

LRN

2013-07-29 11:00:08 UTC

Post by NightStrike

Right -- but *some* executables would need to actually BE different, aside
from the underlying posix library's behavioral changes, to get a "real" MSYS
environment.

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
- make w/ DOS path handling

Yes. Note that the code DOES exist in make, but is disabled when make is
built for Cygwin. One might question whether it was a good idea to
disable support for W32 paths in make that runs on W32 (with a
compatibility layer, but still...) in the first place.

Post by Corinna Vinschen
- gcc targeting Mingw rather than Cygwin.

You already have that, it's called "mingw cross-compiler for cygwin".
And that is not what msys users use.
I think you've meant something different here, i'm not sure what.

Post by Corinna Vinschen
- Cygwin compiler provided as cross tool $arch-pc-cygwin-gcc, etc.(*)

Yes. That includes binutils too (obviously)

Also:

base-files:
add /mingw/bin to PATH when running in MINGW mode
(adding /mingw/share/{man,info} to {MAN,INFO}PATH is also useful, but
not really mandatory)
(ok, these aren't binaries, so maybe they don't qualify)

bash:
Fix various instances of '\r' handling.

coreutils:
Bring back the '\r' support in md5sum. This is another change that might
just go into Cygwin in general. I'm not sure whether '\r' is a valid
character to use in filenames.
(optionally) embed install.exe.manifest into install.exe

There are also patches to rebase that i find useful, but they are up to
debate.

Yeah, that list turned out to be really short. There may be other
packages that need fixing ('\r' handling is the likely culprit), but i
haven't bumped into them yet, so i wouldn't know.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Corinna Vinschen

2013-07-29 11:18:56 UTC

Post by NightStrike
Perhaps it would be useful to actually identify which packages have
extenuating needs. Maybe it's just one or two. Maybe it's all but
one or two. I don't think that currently, the problem space is
properly enumerated, but is instead living in the abstract.

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
- make w/ DOS path handling

Cygwin paths can contain "x:" as part of a normal filename.

Post by Corinna Vinschen
- gcc targeting Mingw rather than Cygwin.

You already have that, it's called "mingw cross-compiler for cygwin".
And that is not what msys users use.
I think you've meant something different here, i'm not sure what.

That's exactly what I meant. Of course we have a mingw cross compiler
in the Cygwin distro, but as far as the discussion on the mingw-w64 list
showed, MSYS users apparently prefer the "native" gcc compiler (the one
called "gcc") to produce mingw executables (aka "native Windows
exectables running without a compat layer") to avoid cross compiling
when creating native Windows executables. If the native gcc in an MSYS
install targets MSYS, and if you had to use a cross compiler to create
native Windows executables (as in Cygwin), there would ne no reason for
MSYS at all since it would be equivalent to Cygwin.

Post by Corinna Vinschen
- Cygwin compiler provided as cross tool $arch-pc-cygwin-gcc, etc.(*)

Yes. That includes binutils too (obviously)
add /mingw/bin to PATH when running in MINGW mode
(adding /mingw/share/{man,info} to {MAN,INFO}PATH is also useful, but
not really mandatory)
(ok, these aren't binaries, so maybe they don't qualify)
Fix various instances of '\r' handling.
Bring back the '\r' support in md5sum. This is another change that might
just go into Cygwin in general. I'm not sure whether '\r' is a valid
character to use in filenames.

Yes.

Post by LRN
(optionally) embed install.exe.manifest into install.exe
There are also patches to rebase that i find useful, but they are up to
debate.
Yeah, that list turned out to be really short. There may be other
packages that need fixing ('\r' handling is the likely culprit), but i
haven't bumped into them yet, so i wouldn't know.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

LRN

2013-07-29 15:36:13 UTC

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
- gcc targeting Mingw rather than Cygwin.

You already have that, it's called "mingw cross-compiler for cygwin".
And that is not what msys users use.
I think you've meant something different here, i'm not sure what.

"native" compiler is the one that does not do cross-compiling. That is,
it compiles with $build==$host, produces code for $build, and looks for
headers/libs in $prefix/{include,lib} (unlike cross-compilers, which
produce code for $host!=$build and look in $prefix/$host/{include,lib}).
Whether it's called "gcc.exe" or "i686-w64-mingw32.exe" is not important
(well, it is, but usually you just symlink gcc.exe to i686-w64-mingw32.exe).

Post by Corinna Vinschen
to produce mingw executables (aka "native Windows
exectables running without a compat layer") to avoid cross compiling
when creating native Windows executables. If the native gcc in an MSYS
install targets MSYS, and if you had to use a cross compiler to create
native Windows executables (as in Cygwin), there would ne no reason for
MSYS at all since it would be equivalent to Cygwin.

Yes. In this case i don't see how Cygwin fits in here. Unless you
suddenly decided to become a MinGW toolchain vendor.
/usr/bin/gcc is a cygwin gcc that targets cygwin
/mingw/bin/gcc is MinGW gcc that targets W32.

Anything that resides in /mingw is completely outside of cygwin domain,
which is why i was surprised to hear that you wanted to provide mingw gcc.

I would expect people to get cygwin/msys in one place, and get MinGW in
another. Even mingw-get, while being a source of both msys and mingw
packages, clearly distinguishes betweent he two.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Corinna Vinschen

2013-07-29 15:47:25 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
- gcc targeting Mingw rather than Cygwin.

You already have that, it's called "mingw cross-compiler for cygwin".
And that is not what msys users use.
I think you've meant something different here, i'm not sure what.

I'm perfectly aware what a cross compiler is, but in case of Cygwin or
MSYS you have a bit of a hard time to define what a native compiler is,
the one creating Cygwin binaries or the one creating Windows binaries
with the unspoken assumption that they don't require the Cygwin DLL.
That's why I tried to describe it in so may word and apparently failed.

Yes. In this case i don't see how Cygwin fits in here. Unless you
suddenly decided to become a MinGW toolchain vendor.
/usr/bin/gcc is a cygwin gcc that targets cygwin
/mingw/bin/gcc is MinGW gcc that targets W32.
Anything that resides in /mingw is completely outside of cygwin domain,
which is why i was surprised to hear that you wanted to provide mingw gcc.
I would expect people to get cygwin/msys in one place, and get MinGW in
another. Even mingw-get, while being a source of both msys and mingw
packages, clearly distinguishes betweent he two.

That's what I understood differently. From the discussion on mingw-w64
it seemed that a mingw dev using Cygwin/MSYS would prefer if the default
gcc creates non-CYgwin/MSYS, but rather Windows-only binaries.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Charles Wilson

2013-07-29 16:37:43 UTC

Post by Corinna Vinschen
That's what I understood differently. From the discussion on mingw-w64
it seemed that a mingw dev using Cygwin/MSYS would prefer if the default
gcc creates non-CYgwin/MSYS, but rather Windows-only binaries.

I agree with Corinna here. I think LRN is assuming that the
installation structure will remain the same as MinGW/MSYS going forward,
and I do not believe that is correct -- at least, that's not what
Corinna is proposing IIUC.

Here's how MinGW/MSYS works right now:

*normally* the batch file you use to enter the "MinGW" environment,
launches bash with
1) MSYSTEM=MINGW32
2) /mingw/bin at the "front" of $PATH (wherever your mount
table happens to say that is)

Your phyiscal directory structure *normally* looks something like this:

C:\MinGW <<<<< mounted as /mingw
bin MinGW (that is, "native" win32) gcc.exe & friends
include "native" win32 headers
lib and libraries
msys
1.0 <<<<< mounted as /
bin gcc.exe -- only this time, it's an
msys-target compiler. But note that it is
actually configured as a "native" compiler
include msys headers and
lib libraries

Because of #1 and #2 above, uname reports MINGW32, and the first gcc
found in your path is the "native" win32 MinGW one.

However, an *msys* developer would launch a script that invokes bash with
1) MSYSTEM=MSYS
2) /bin at the "front" of $PATH

In this case, uname reports MSYS, and the first gcc found in your path
is the msys one.

=====-----=====

In the new scenario, we might have a separable installation -- maybe
c:\msys\2.0\
but there isn't any gcc.exe installed there. And then you might install
MinGW gcc somewhere like
c:\MinGW-4.8.1\
and just arrange that
c:\msys\2.0\etc\fstab
has
c:\MinGW-4.8.1\ /mingw

and again, you make sure that /mingw/bin is in your $PATH.

However, in the new scenario, you MIGHT have, in
c:\msys\2.0\i686-pc-cygwin\, a cross compiler targetting "cygwin/msys"
and running on...MinGW (even though "MinGW" environment is, for all
intents and purposes, a slightly modified cygwin -- but uname reports
MINGW32 just like it does for "native" MinGW/win32 operation). To build
msys apps in this environment, you have to use --host=i686-pc-cygwin
(and remember, because uname is reporting "MINGW32", any build system
will operate under the assumption that you are, in fact, cross compiling).

There are some advantages to the former system, not least of which is
that when MSYSTEM=MSYS, you're compiling natively so you can easily run
any test suites without having to play games with the build system.

I think LRN is assuming that the gcc installed in /bin would be the
cygwin gcc (configured as a native compiler), and we'd continue to play
$MSYSTEM/$PATH games.

One additional "advantage" to the former system is the autotools. Right
now we can have a "clean" separation between aclocal/.m4 files that have
data corresponding to MinGW-compiled native libs and tools, and
aclocal/.m4 files that have data corresponding to the msys ones --
because we have two entirely distinct "sets" of autotools.

/mingw/* has the whole panoply of autoconf2.1/2.5/wrapper,
automake1.4--1.12/wrapper, libtool, gettext, and libintl. All are
configured with --prefix=/mingw, so they look in /mingw/share/aclocal*/
for .m4 stuff.

OTOH, in /{bin,lib,share} we have one specific version of autoconf (2.59
IIRC), one specific version of automake (1.11?), libtool specially
hacked to support msys (because "regular" libtool does not), gettext,
and libintl. Because msys has never been, and was not intended to be, a
public "triple" value, these versions' config.guess/config.sub were
modified to recognize the MSYS uname, and report i686-pc-msys as a
triple, and to actually handle that triple correctly.

The /mingw version of the autotools was not hacked in this way.

This way, when porting some package in /mingw land and running
re-autoconf, users don't get "fooled" by picking up stuff in
/share/aclocal/* -- they only "see" the .m4 files that correspond to
OTHER fully-ported-to-mingw libs and tools.

--
Chuck

LRN

2013-07-30 01:18:10 UTC

I agree with Corinna here. I think LRN is assuming that the
installation structure will remain the same as MinGW/MSYS going forward,
and I do not believe that is correct -- at least, that's not what
Corinna is proposing IIUC.
In the new scenario, we might have a separable installation -- maybe
c:\msys\2.0\
but there isn't any gcc.exe installed there. And then you might install
MinGW gcc somewhere like
c:\MinGW-4.8.1\
and just arrange that
c:\msys\2.0\etc\fstab
has
c:\MinGW-4.8.1\ /mingw
and again, you make sure that /mingw/bin is in your $PATH.
However, in the new scenario, you MIGHT have, in
c:\msys\2.0\i686-pc-cygwin\, a cross compiler targetting "cygwin/msys"
and running on...MinGW (even though "MinGW" environment is, for all
intents and purposes, a slightly modified cygwin -- but uname reports
MINGW32 just like it does for "native" MinGW/win32 operation). To build
msys apps in this environment, you have to use --host=i686-pc-cygwin
(and remember, because uname is reporting "MINGW32", any build system
will operate under the assumption that you are, in fact, cross compiling).

That might work. It also means that you'll need a new cygwin
cross-compiler (a mingw->cygwin cross-compiler, although in reality it
might be cygwin->cygwin native compiler that looks like a
cross-compiler). Unless existing one can be stuffed into
/usr/i686-pc-cygwin without any ill effects.

Post by Charles Wilson
There are some advantages to the former system, not least of which is
that when MSYSTEM=MSYS, you're compiling natively so you can easily run
any test suites without having to play games with the build system.
I think LRN is assuming that the gcc installed in /bin would be the
cygwin gcc (configured as a native compiler), and we'd continue to play
$MSYSTEM/$PATH games.

Yes, LRN is assuming that /usr/bin/gcc is a cygwin-gcc, and
/mingw/bin/gcc is a mingw gcc.
$MSYSTEM/$PATH games worked well enough so far.

Post by Charles Wilson
One additional "advantage" to the former system is the autotools. Right
now we can have a "clean" separation between aclocal/.m4 files that have
data corresponding to MinGW-compiled native libs and tools, and
aclocal/.m4 files that have data corresponding to the msys ones --
because we have two entirely distinct "sets" of autotools.
/mingw/* has the whole panoply of autoconf2.1/2.5/wrapper,
automake1.4--1.12/wrapper, libtool, gettext, and libintl. All are
configured with --prefix=/mingw, so they look in /mingw/share/aclocal*/
for .m4 stuff.

Yes, that is a good point.

Post by Charles Wilson
OTOH, in /{bin,lib,share} we have one specific version of autoconf (2.59
IIRC), one specific version of automake (1.11?), libtool specially
hacked to support msys (because "regular" libtool does not), gettext,
and libintl. Because msys has never been, and was not intended to be, a
public "triple" value, these versions' config.guess/config.sub were
modified to recognize the MSYS uname, and report i686-pc-msys as a
triple, and to actually handle that triple correctly.

Yes, but with cygwin that is going to go away, as i686-pc-cygwin IS a
valid triplet. More reasons to not to let mingw apps see these autotools
and m4 files.

Post by Charles Wilson
The /mingw version of the autotools was not hacked in this way.

Though it might be hacked in other, unspecified ways (for example, i
mess with stuff a lot to integrate W32 CPython; this is not needed for
msys-python).

You've laid out advantages and "advantages" of the former system.
What are the advantages of the latter system?

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

LRN

2013-07-30 00:45:10 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Very good point. This would perhaps show us much better where we're
heading here. From the current input I only see the following required
- gcc targeting Mingw rather than Cygwin.

You already have that, it's called "mingw cross-compiler for cygwin".
And that is not what msys users use.
I think you've meant something different here, i'm not sure what.

to produce mingw executables (aka "native Windows
exectables running without a compat layer") to avoid cross compiling
when creating native Windows executables. If the native gcc in an MSYS
install targets MSYS, and if you had to use a cross compiler to create
native Windows executables (as in Cygwin), there would ne no reason for
MSYS at all since it would be equivalent to Cygwin.

Yes. In this case i don't see how Cygwin fits in here. Unless you
suddenly decided to become a MinGW toolchain vendor.
/usr/bin/gcc is a cygwin gcc that targets cygwin
/mingw/bin/gcc is MinGW gcc that targets W32.
Anything that resides in /mingw is completely outside of cygwin domain,
which is why i was surprised to hear that you wanted to provide mingw gcc.
I would expect people to get cygwin/msys in one place, and get MinGW in
another. Even mingw-get, while being a source of both msys and mingw
packages, clearly distinguishes betweent he two.

The problem is not that you can't have a W32-targetted gcc.exe in /usr/bin.

The problem is that the convention that everyone has been following for
years now is that all mingw stuff lives in /mingw, and all msys (cygwin)
stuff lives in /usr. These two are never mixed.

If you start providing alternative gccs, that might confuse people. A
lot. Since for people who installed cygwin-gcc, gcc.exe will produce
cygwin binaries, while the ones with mingw-gcc will have a gcc.exe that
produces mingw binaries. Also, how will you have both cygwin-gcc and
mingw-gcc installed at the same time (some people do need that)?

Relying on prefixes (i686-pc-cygwin-gcc and i686-w64-mingw32-gcc) might
work, but not all packages are prefix-aware.
This also brings us back to the distinction between native and
non-native toolchains. To have both gccs in /usr, you'll have to make
them look for stuff in /usr/$target/{include,lib}. Can't have both mingw
and cygwin libs/headers in /usr/{lib,include}, obviously.
While that might work, i'd guess that it will also break in a some
cases, where people (and scripts) expect that $prefix/{lib,include} are
the right dirs to use.

If it looks in $prefix/$target/{lib,include} and has a name $target-gcc,
then from user's point of view it's a cross-compiler (even if has
build==host==target), and we've already decided that you already have a
cross-compiler and don't need another one.

/mingw also provided a way to swap mingw subsystem (just edit /etc/fstab
to make /mingw point to a different dir, then restart the shell). With
everything living in /usr that might be unnecessary, but now that i
think of this...how are you going to separate compiled binaries from
each other? Say, your /usr/bin/i686-w64-mingw-gcc builds libxml2-2.dll.
That lands in /usr/bin. And when you compile libxml with
/usr/bin/x86_64-w64-mingw-gcc, it builds libxml2-2.dll, and it lands
into the same directory. Where else could it go? /usr/i686-w64-mingw/bin
or /usr/x86_64-w64-mingw/bin? I've never seen mingw doing that kind of
stuff. And how are you going to maintain PATH for them?
If you propose to have only the toolchain in /usr, but build&install
everything with --prefix=/mingw...that, again, might work in some cases,
but not the others. Also, which packages are "the toolchain"? You will
also need to configure gcc that lives in /usr to look for headers/libs
in /mingw/{include,lib}.

So i'd suggest to stick with /usr and /mingw convention and let mingw
take care of itself. It's simpler that way.

- --
O< ascii ribbon - stop html email! - www.asciiribbon.org

Corinna Vinschen

2013-07-30 09:04:28 UTC

Post by LRN
I would expect people to get cygwin/msys in one place, and get MinGW in
another. Even mingw-get, while being a source of both msys and mingw
packages, clearly distinguishes betweent he two.

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.

So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

Are you going to help implementing this? If not, I'm missing input
from any of the developers working on MSYS2...

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Alexey Pavlov

2013-07-30 09:32:09 UTC

2013/7/30 Corinna Vinschen

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.
So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

As I see all discussion not about implementing but about philosophy of
MSYS. At the start of discussion I wrote about my changes in Cygwin
sources to have MSYS. And also send small patches but nothing really
doing in this direction.
What steps do we need to start any work on implementing it?
Now I see next points where we can change Cygwin functionality:
1. uname function
2. reading /etc/fstab
3. passing arguments and environment variables to non-Cygwin
processes ( environ.cc, spawn.cc )
4. symlinks changes (copy instead symlink)

Post by Corinna Vinschen
Are you going to help implementing this? If not, I'm missing input
from any of the developers working on MSYS2...

I'm on vacation until September. But I can help if any work starting
in this direction.

Regards,
Alexey.

Post by Corinna Vinschen
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Corinna Vinschen

2013-07-30 09:47:02 UTC

Post by Alexey Pavlov
2013/7/30 Corinna Vinschen

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.
So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

As I see all discussion not about implementing but about philosophy of
MSYS.

No. We were talking about how to implement the changes. Your patches
change Cygwin directly, but the idea is to keep the actual changes
separate, outside of Cygwin, as hooks.

Post by Alexey Pavlov
At the start of discussion I wrote about my changes in Cygwin
sources to have MSYS. And also send small patches but nothing really
doing in this direction.
What steps do we need to start any work on implementing it?
1. uname function
2. reading /etc/fstab
3. passing arguments and environment variables to non-Cygwin
processes ( environ.cc, spawn.cc )
4. symlinks changes (copy instead symlink)

Yes, these are the behavioral changes you want to implement, but this is
the discussion as to *how* to implement them. You never actually took
part in this discssion yet.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Alexey Pavlov

2013-07-30 10:27:20 UTC

Post by Alexey Pavlov
2013/7/30 Corinna Vinschen

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.
So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

As I see all discussion not about implementing but about philosophy of
MSYS.

No. We were talking about how to implement the changes. Your patches
change Cygwin directly, but the idea is to keep the actual changes
separate, outside of Cygwin, as hooks.

Yes, these are the behavioral changes you want to implement, but this is
the discussion as to *how* to implement them. You never actually took
part in this discssion yet.

Now there are not many changes that I do in Cygwin sources and it
change only small functionality without breaking anything for cygwin
users (only symlinks is breaking thing).
If we need to implement changes in external DLL we doesn't want to
full rewrite modified functions in this DLL. External DLL need only do
changes inside the Cygwin functions not full replace this functions I
think.
Maybe you create separate branch in CVS source where we can try to
implement MSYS mode and also create hook in uname function, for
example, to try how it work?
From my POV we need to create separate distro for MSYS where in bash
profile we can activate MSYS mode for Cygwin.dll with some environment
variable that tell to load msys.dll. But if you want has working
applications between normal Cygwin and msys-in-cygwin then they need
to be linked only with cygwin.dll and doesn't need to know anything
about msys.dll.

Post by Corinna Vinschen
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Corinna Vinschen

2013-07-30 10:33:50 UTC

Post by Alexey Pavlov
2013/7/30 Corinna Vinschen

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.
So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

As I see all discussion not about implementing but about philosophy of
MSYS.

No. We were talking about how to implement the changes. Your patches
change Cygwin directly, but the idea is to keep the actual changes
separate, outside of Cygwin, as hooks.

Yes, these are the behavioral changes you want to implement, but this is
the discussion as to *how* to implement them. You never actually took
part in this discssion yet.

That was part of cgf's original proposal. We have the CW_HOOK mechanism
to overload entire functions (per the discussion not fully implemented
yet) but there's no problem to implement a kind of mini-hook with a
set of function pointers called from the CYgwin DLL at certain, yet to
be defined points.

Post by Alexey Pavlov
Maybe you create separate branch in CVS source where we can try to
implement MSYS mode and also create hook in uname function, for
example, to try how it work?

That should be possible.

Post by Corinna Vinschen
From my POV we need to create separate distro for MSYS where in bash

profile we can activate MSYS mode for Cygwin.dll with some environment
variable that tell to load msys.dll. But if you want has working
applications between normal Cygwin and msys-in-cygwin then they need
to be linked only with cygwin.dll and doesn't need to know anything
about msys.dll.

Exactly.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Alexey Pavlov

2013-07-30 10:59:40 UTC

Post by Alexey Pavlov
2013/7/30 Corinna Vinschen

Fine with me, of course. I don't have that problem myself so whatever
works better for you is ok. But then the number of changed packages
doesn't really matter. From my POV MSYS sticks to being it's own distro
with another focus than the Cygwin distro.
So we're back to discussing in how far MSYS can be implemented using a
stock Cygwin DLL under the hood and the tweaks being in an external MSYS
DLL so users can mix the best of both worlds as they see fit for their
purpose.

As I see all discussion not about implementing but about philosophy of
MSYS.

No. We were talking about how to implement the changes. Your patches
change Cygwin directly, but the idea is to keep the actual changes
separate, outside of Cygwin, as hooks.

Yes, these are the behavioral changes you want to implement, but this is
the discussion as to *how* to implement them. You never actually took
part in this discssion yet.

We need to determine places where you need to create hooks in
Cygwin.dll and also prototypes for external functions (input and
output arguments). Msys mode only change functions for non-Cygwin
applications and in this case where it can be handled? In cygwin.dll
or in msys.dll?
For example, when we pass arguments to new process we determine in
Cygwin is new process depends on cygwin.dll or not and make decision
about changing arguments. If new process is non-cygwin then we try to
change arguments else we leave them as is. If we need do implement
decision logic in msys.dll then we need to rewrite many code. But if
we implement it in Cygwin dll then it was simple like this:
if (!iscygexec() && msysmode)
{
// hook to function from msys.dll
} else {

// default cygwin code
}

In this case we can create mini-hooks in cygwin DLL that call
functions from msys.dll if necessary.

Post by Alexey Pavlov
Maybe you create separate branch in CVS source where we can try to
implement MSYS mode and also create hook in uname function, for
example, to try how it work?

That should be possible.

Post by Corinna Vinschen
From my POV we need to create separate distro for MSYS where in bash

Exactly.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Christopher Faylor

2013-07-30 14:55:33 UTC

Post by Alexey Pavlov
We need to determine places where you need to create hooks in
Cygwin.dll and also prototypes for external functions (input and
output arguments). Msys mode only change functions for non-Cygwin
applications and in this case where it can be handled? In cygwin.dll
or in msys.dll?

This is the outstanding question for the last month or so.

Post by Alexey Pavlov
For example, when we pass arguments to new process we determine in
Cygwin is new process depends on cygwin.dll or not and make decision
about changing arguments. If new process is non-cygwin then we try to
change arguments else we leave them as is. If we need do implement
decision logic in msys.dll then we need to rewrite many code. But if

No. In the spawn function, you have something like:

if (!callout (CO_SPAWN, mode, &argv, &argc))
...

And a DLL which has been preloaded and registered with cygwin1.dll will
modify the arguments as needed.

Either that or, the other proposal is that MSYS could just use LD_PRELOAD
but, as attractive as the notion of not modifying Cygwin at all might be,
it would mean that the MSYS dll would have to do a lot more work for
the above scenario because it would have to emulate a lot of what Cygwin
does to emulate exec/spawn.

So, in the interests of moving this discussion along, here is a more
fleshed out proposal:

Any helper DLL would be registered using a CYGWIN environment variable
preload keyword:

set CYGWIN=preload:msys.dll

An error will be issued if Cygwin can't find the preload dll.

The DLL will register callouts via:

cygwin_internal (CW_CALLOUT, function_which_handles_callouts);

Callout Arguments
CO_SPAWN mode (value), argv (reference), argc (reference)
CO_UNAME utsname (reference)
CO_ENV envp (reference)
CO_SYMLINK is_native_symlink (value), oldpath (reference), newpath (reference)*
CW_MALLOC Amount of cygheap memory to allocate

*Choosing where to call this might be tricky. I don't see any reason
why MSYS shouldn't avail itself of the new native symlinks if they
can be created. Otherwise, symlink would have to be short-circuited
in some cases so maybe we need to check the return value of the callout
function:

...
switch (callout (CW_SYMLINK, is_native_symlink, &oldpath, &newpath))
{
case CO_R_KEEP_GOING:
break;
case CO_R_SHORT_CIRCUIT:
return 0;
case CO_R_ERR:
return -1;
}
...

The callout function will look something like:

enum callout_return
callout (enum callout co, ...)
{
if (!callout_func)
return CO_R_KEEP_GOING;
va_list ap;
va_start (ap, co);
return callout_func (co, ap);
}

(I have to think about the most efficient way to implement this)

What other CO_* values are needed?

cgf

Christopher Faylor

2013-07-30 15:43:49 UTC

This is the outstanding question for the last month or so.

if (!callout (CO_SPAWN, mode, &argv, &argc))
...
And a DLL which has been preloaded and registered with cygwin1.dll will
modify the arguments as needed.
Either that or, the other proposal is that MSYS could just use LD_PRELOAD
but, as attractive as the notion of not modifying Cygwin at all might be,
it would mean that the MSYS dll would have to do a lot more work for
the above scenario because it would have to emulate a lot of what Cygwin
does to emulate exec/spawn.
So, in the interests of moving this discussion along, here is a more
Any helper DLL would be registered using a CYGWIN environment variable
set CYGWIN=preload:msys.dll
An error will be issued if Cygwin can't find the preload dll.
cygwin_internal (CW_CALLOUT, function_which_handles_callouts);
Callout Arguments
CO_SPAWN mode (value), argv (reference), argc (reference)
CO_UNAME utsname (reference)
CO_ENV envp (reference)
CO_SYMLINK is_native_symlink (value), oldpath (reference), newpath (reference)*
CW_MALLOC Amount of cygheap memory to allocate

I just implemented some of this and I'm wondering if we might need
some way of controlling the flow on entry to a function and on exit.
So maybe the above could be implemented as:

CO_UNAME_ENTRY
CO_UNAME_EXIT

but that might be overkill. It wouldn't necessarily make sense for
CO_SPAWN either.

cgf

Christopher Faylor

2013-07-30 16:14:20 UTC

Post by Christopher Faylor
CW_MALLOC Amount of cygheap memory to allocate

I didn't make it clear that this would be a cygwin_internal() call.

cgf

Earnie Boyd

2013-07-29 11:19:42 UTC

Post by Corinna Vinschen
- make w/ DOS path handling

MSYS has never modified the make source and it uses the CYGWIN coding.
The only thing is we do accept windows pathing in the form of
C:/SOME/DIR but not C:\SOME\DIR.

The only thing I know that is currently changed is bash to add a -W
option to pwd. That should be carried forward to /bin/pwd but that
currently isn't the case.

All of the path handling magic happens in msys-1.0.dll. The other
thing that happens are chosen sane defaults for the options provided
by CYGWIN so that users reporting errors wouldn't have the chance to
screw with a known set of configuration and the CYGWIN variable is
thus ignored. And all file handles are set to _O_BINARY without the
ability to change it; I don't know if that matters for the current
version of Cygwin but mentioning it just in case.

--
Earnie
-- https://sites.google.com/site/earnieboyd

Charles Wilson

2013-07-29 12:20:31 UTC

Post by Earnie Boyd

Post by Corinna Vinschen
- make w/ DOS path handling

MSYS has never modified the make source and it uses the CYGWIN coding.
The only thing is we do accept windows pathing in the form of
C:/SOME/DIR but not C:\SOME\DIR.

No.

Here's the patches applied to msys's make-3.81

make/01-make-3.81-2-cygwin.patch -- from cygwin's 3.81-2 package
make/02-make-3.81-dos-path.patch
make/03-make-3.81-case_preserve.patch
make/04-make-3.81-msys.patch

I've archives all the patches I used when I last built the msys apps --
it's over 1MB after xz compression. I'll figure out somewhere to put it
and post the link later today.

Here's the file listing -- 151 different patches to 56 different
products. Some were unofficial, since I never actually published my
msys-tcl port.

expat/expat-2.0.1-1-msys.patch
libiconv/00-wchar-libiconv-1.14.patch
libiconv/01-undo-cygwin17-reloc-libiconv-1.14.patch
libiconv/02-reloc-libiconv-1.14.patch
libiconv/03-cygwin-libiconv-1.14.patch
libiconv/libiconv-1.13.1-2-msys.patch
libiconv/libiconv-1.14-1.msys.patch
libiconv/libiconv-1.14-1.src.patch
autoconf/autoconf-2.67-1-msys.patch
autoconf/autoconf-2.68-1.msys.patch
autoconf/autoconf-2.68-1.src.patch
autogen/autogen-5.10.1-1-msys.patch
automake/automake-1.11.1-1-msys.patch
bash/01-bash-3.1-msys.patch
bison/bison-2.4.2-1-msys.patch
bzip2/01-cygming-bzip2-1.0.6.src.patch
bzip2/bzgrep-debian-1.0.5-6.patch
bzip2/bzip2-1.0.4-bzip2recover.patch
bzip2/bzip2-1.0.6-1.msys.patch
bzip2/bzip2-1.0.6-1.src.patch
bzip2/bzip2-1.0.6-3.mgw.patch
bzip2/bzip2-1.0.6-progress.patch
coreutils/01-coreutils-5.97-1-cygwin.patch
coreutils/02-coreutils-5.97-3-msys.patch
coreutils/03-coreutils-5.97-2-autoreconf.patch
coreutils/04-coreutils-5.97-3-more-autoreconf.patch
crypt/crypt-1.1_1-3-msys.patch
cvs/9991-cvs-1.12.13-10-cygwin-src.patch
cvs/9992-cvs-1.12.13-10-cygwin-ext.patch
cvs/9993-cvs-1.12.13-2-msys.patch
cvs/9994-cvs-1.12.13-po-template.patch
cygutils/cygutils-1.3.4-3-msys-unmerged.patch
cygutils/cygutils-1.3.4-4-msys.patch
dash/01-cygwin-dash-0.5.5.1-2.patch
dash/02-msys-dash-0.5.5.1.patch
diffutils/01-diffutils-2.8.7-3-gnulib.patch
diffutils/02-diffutils-2.8.7-3-msys.patch
libtool/0001-Fix-regression-in-command-line-length-computation.patch
libtool/0002-cygwin-mingw-Create-UAC-manifest-files.patch
libtool/0003-Pass-various-runtime-library-flags-to-GCC.patch
libtool/0004-Fix-linking-with-fstack-protector.patch
libtool/0005-msys-support.patch
libxml2/2.7.3-doc-install.patch
libxml2/libxml2-2.7.6-1-msys.patch
lndir/lndir-1.0.1-2-msys.patch
m4/01-m4-1.4.14-1-msys.patch
make/01-make-3.81-2-cygwin.patch
make/02-make-3.81-dos-path.patch
make/03-make-3.81-case_preserve.patch
make/04-make-3.81-msys.patch
man/01-man-1.6e-1-cygwin-src-fwdprt.patch
man/02-man-1.6f-2-msys.patch
minires/minires-1.01-1.patch
minires/minires-1.02_1-2-msys.patch
mktemp/mktemp-1.6-2-msys.patch
msysCORE/foo.patch
msysCORE-old/all-deltas.patch
msysCORE-old/build-machinery-1.0.15.patch
msysCORE-old/pseudo-reloc-fork-fix.patch
openssh/01-openssh-5.4p1-msys.patch
openssl/01-msys-initial.patch
openssl/openssl-0.9.6-x509.patch
openssl/openssl-0.9.7-beta5-version-add-engines.patch
openssl/openssl-0.9.8e-crt.patch
patch/9991-patch-2.5.8-9-cygwin-src-fwdprt.patch
patch/9992-patch-2.6.1-1-msys.patch
patch/msys-build-patch
perl-old/perl-5.6.1_2-1-msys.patch
popt/01-popt-0.15-2-msys.patch
rebase/rebase-4.0.1_1-1.msys.patch
rebase/rebase-4.0.1_1-1.src.patch
rsync/01-rsync-3.0.8-msys.patch
rxvt/991-rxvt-2.7.2-msys-dll-name.patch
rxvt/900-rxvt-2.7.2-yodl.patch
rxvt/901-rxvt-2.7.2-doc.patch
rxvt/902-rxvt-2.7.2-changelogs.patch
rxvt/903-rxvt-2.7.2-remove-files.patch
rxvt/904-rxvt-2.7.2-msys-orig.patch
rxvt/905-rxvt-2.7.2-license.patch
rxvt/990-rxvt-2.7.2-msys-3.patch
rxvt/992-rxvt-2.7.2-msys-fixup-resources.patch
sed/sed-4.2.1-2-msys.patch
tar/990-tar-1.22-1-cygwin-fwdport.patch
tar/991-tar-1.23-1-msys.patch
tar/992-silence-record-size-warning.patch
tcl/03-cygwin-un-win32ify.patch
tcl/04-cygwin-env-sync.patch
tcl/05-msys-libprefix-load.patch
tcl/06-cygwin-nocolons.patch
tcl/07-msys-termio-without-parext.patch
tcl/08-msys-workaround-stdhandles-ebadf.patch
tcl/09-cygwin-use-autoimport.patch
tcl/10-cygwin-msys-libprefix-build.patch
tcl/11-msys-tcl-m4.patch
tcl/8.5-cygwin.patch
tcl/8.5.10-tea-m4.patch
tcl/tcl-8.5.11-1.msys.patch
termcap/termcap-0.20050421_1-2-msys.patch
texinfo/9991-texinfo-4.13-3-cygwin-src.patch
texinfo/9992-texinfo-4.13a-1-msys-gnulib-updates.patch
texinfo/9993-texinfo-4.13a-2-msys-other.patch
texinfo/9994-texinfo-4.13a-3-msys-libintl.patch
unzip/01-unzip-6.0-10-cygwin.patch
unzip/02-unzip-6.0-1-msys.patch
unzip/bzip2-1.0.5-10.partial-src.patch
vim/vim-7.2-2-msys.patch
wget/01-wget-1.12-1-msys.patch
xz/01-cygwin-xz-5.0.2-1.patch
xz/02-msys-missing-stdint-inttypes.patch
xz/03-msys-getopt-workaround.patch
xz/04-msys-other.patch
xz/xz-5.0.3-1.msys.patch
xz/xz-rollup-fixes-7fcc6334.patch
zip/bzip2-1.0.5-10.partial-src.patch
zip/zip-3.0-1-msys.patch
zip/zip-3.0-11.src.patch
zip/zip-3.0-build.patch
file/9991-file-5.00-3-cygwin-src-partial.patch
file/9992-file-5.04-1-msys.patch
findutils/01-findutils-4.4.0-3-cygwin-src-fwdprt.patch
findutils/02-findutils-4.4.2-2-msys.patch
flex/flex-2.5.35-2-msys.patch
flex/flex-debian-2.5.35-7-partial.patch
gawk/999-gawk-3.1.7-2-msys.patch
gdbm/gdbm-1.8.3-3-msys.patch
gettext/gettext-0.18.1.1-1.msys.patch
gettext/gettext-0.18.1.1-1.src.patch
gettext/00-slowdown-cvs-archive-creation.patch
gettext/01-msys-fixes.patch
gettext/02-locale-gettext-0.18.1.1.patch
gettext/03-cygwin-gettext-0.18.1.1.patch
gettext/04-mingw-script-slash-fix.patch
gmp/gmp-5.0.1-1-msys.patch
grep/999-grep-2.5.4-2-msys.patch
groff/01-doc-gfdl-msys.patch
groff/02-msys-install.patch
guile/guile-1.8.7-2-msys.patch
gzip/9991-grep-1.3.12-2-cygwin-src.patch
gzip/9992-grep-1.3.12-2-msys.patch
inetutils/01-buildsystem-updates.patch
inetutils/02-gnulib_20100205-updates.patch
inetutils/03-autoreconf-updates.patch
inetutils/04-documentation-updates.patch
inetutils/05-missing-headers.patch
inetutils/06-cygwin-1.7-1pre-unofficial.patch
inetutils/07-msys-missing-headers.patch
inetutils/08-msys-fixups.patch
inetutils/09-msys-replace-rfunc-for-vista.patch
less/01-434417-LESS_IS_MORE.patch
less/02-less-429-1-cygwin-src.patch
libarchive/01-libarchive-msys-initial.patch

This also misses the changes to the build configurations (e.g. msys
builds foo with --disable-bar, while cygwin uses --enable-bar, etc).

--
Chuck

Corinna Vinschen

2013-07-29 12:49:08 UTC

Post by Earnie Boyd

Post by Corinna Vinschen
- make w/ DOS path handling

MSYS has never modified the make source and it uses the CYGWIN coding.
The only thing is we do accept windows pathing in the form of
C:/SOME/DIR but not C:\SOME\DIR.

The question is, are they really necessary? I seriously doubt that you
have to patch inetutils beyond what's required for Cygwin, unless you
want the kitchen sink to work with DOS paths.

Corinna

--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat

Charles Wilson

2013-07-29 14:22:40 UTC

Post by Corinna Vinschen
The question is, are they really necessary? I seriously doubt that you
have to patch inetutils beyond what's required for Cygwin, unless you
want the kitchen sink to work with DOS paths.

They *were* necessary for msys-1, given the following:

1) msys itself was very old, and had only cygwin-1.3.4 era capabilities
2) the msys targetting compiler is to this day ALSO very old: it's based
on gcc-3.4.4 if you can believe it.

Now, it is certainly true that in the new "era" with msys-on-cygwin,
point #1 above is no longer applicable. And, because there really won't
be an msys target compiler, just the plain old cygwin one (with -DMSYS
[1] and, depending on how the msys hook dll gets loaded, perhaps
-lmsys), then msys apps no longer have to work around old-gcc breakage.

But there are still some patches that will be necessary, and I'm afraid
only experience will tell us which ones. My set of 151 patches to 56
packages [2] is probably the "outer limit"; some packages might be
workable with no special msys changes, others might only need a small
subset of what was previously necessary.

Let's go back to our favorite example, msys-make:

make/01-make-3.81-2-cygwin.patch
--------------------------------
from cygwin's 3.81-2 package, so this is not interesting
when we're discussing *differences* between the cygwin and
"msys" versions

make/02-make-3.81-dos-path.patch
--------------------------------
* configure.in (PATH_SEPARATOR_CHAR): Define to the value of
$PATH_SEPARATOR.
* make.h (PATH_SEPARATOR_CHAR): Define only if still undefined.
Normally, it is defined in config.h.
* config/dospaths.m4 <ac_cv_dos_paths>: Define to yes on Cygwin
as well.
* job.c (construct_command_argv_internal) [HAVE_DOS_PATHS]:
Define sh_chars_sh for Windows platforms that emulate Unix.

This *might* be in make-3.82.90, I'm not sure. But for msys we need to
ensure that ac_cv_dos_paths is "yes", regardless of whether it is yes or
no on cygwin.

make/03-make-3.81-case_preserve.patch
--------------------------------
case-insensitive but case-preserving rule matching. Pretty sure "real"
cygwin doesn't want, and doesn't have, this one. Over in mingw/msys
land we went round and round with "csmake" (case sensitive), "cpmake"
(this one), and "regular" make, and finally decided that cpmake had the
fewest drawbacks when working with "native" gcc...

make/04-make-3.81-msys.patch
--------------------------------
build system changes. In configure.in, the original AC_SUBSTs didn't do
the right thing and needed to use a more modern idiom. In Makefile.am,
the test driver section relied on symlinks, which are obviously broken
in "old" msys.

I think we're all pretty much on the same page, that "msys" needs a
special version of make. Does bash need to be different? probably.
Perl...maybe??? inetutils (clients only) -- I think some of these
changes^Wcripplings are necessary, regardless of the "environment"
improvements that modernizing msys and its associated gcc bring. But
it'll just have to be on a case-by-case basis. I've uploaded my entire
msys patch collection [2] in case anyone is really crazy^Winterested...

[1] This will still be needed because of the msys-*behavioral* patches
that are still going to be necessary, in some cases...

[2] http://mingw.cwilson.fastmail.fm/msys-patches.tar.xz

--
Chuck

Larry Hall (Cygwin Developers)

2013-07-29 18:11:25 UTC

Post by Earnie Boyd

Post by Corinna Vinschen
- make w/ DOS path handling

MSYS has never modified the make source and it uses the CYGWIN coding.
The only thing is we do accept windows pathing in the form of
C:/SOME/DIR but not C:\SOME\DIR.

No.
Here's the patches applied to msys's make-3.81

<snip>

Post by Charles Wilson
I've archives all the patches I used when I last built the msys apps -- it's
over 1MB after xz compression. I'll figure out somewhere to put it and post
the link later today.
Here's the file listing -- 151 different patches to 56 different products.
Some were unofficial, since I never actually published my msys-tcl port.

<snip>

Interesting. Assuming a review of all these patches (and any future MSYS or
even Cygwin-specific patch) won't either be jettisoned as no longer needed
or viewed as common functionality desired in the unified MSYS/Cygwin
universe, I'm starting to think the question of whether EXEs are "drop-in"
chameleons for MSYS or CYGWIN is an implementation detail that the user
should not know about or exploit. We don't want users to have to
understand which EXEs are exactly the same code for both MSYS and Cygwin
and which aren't. Worse, we don't want to have to determine which EXE the
user has "dropped-in" when they report an issue. By these statements, I'm
not saying that the "drop-in" capability shouldn't be used where possible.
But if it is used, it should be something entirely opaque to the user.
Even beyond that, we should actively hide this technical "sugar" from the
end users to avoid the potential support nightmare. This shouldn't be a
big deal though. We'd just need to install all the tools for each
environment separately.

--
Larry

Corinna Vinschen

2013-07-29 09:25:13 UTC

Post by Corinna Vinschen
Here's where I disagree. I think the executables should *not* rely on
MSYS.dll being available. Ideally the executables are linked against
the Cygwin DLL and MSYS.dll is called as a side-by-side implementation.
So, if MSYS.dll isn't available, they still function as normal Cygwin
executables. That's why I proposed the solution(s) in
http://cygwin.com/ml/cygwin-developers/2013-07/msg00003.html

Depends. If the Cygwin DLL itself cares for loading a side-by-side
MSYS.dll, then yes. If the crt0.o takes over this job, then only
the applications rebuilt with this crt0.o will do that.

Post by Charles Wilson
Now, back in my MSYS-on-cygwin installation, there are SOME
executables that are actually *different* that the corresponding
ones in real-cygwin-land. Stuff like make, bash, perl, etc -- all
may have been compiled with different options because we (the
mingw/msys people) want them to behave differently, in ways that
can't automatically be handled by the hooked changes in
cygwin1.dll's own behavior.
Have I got that all correct?

More or less, yes, except for the tiny detail above.

Post by Charles Wilson
[1] I think this is better than an environment var, because then my
"regular" cygwin tree and my "msys" cygwin tree would both just
work, without needed extraneous global env vars that might interfere
with the other's operation.
In fact, I might want *different* CYGWIN env var settings for the
two trees, but unless I set them in the global env then
StartMenu-launched apps lose out.

They don't lose if you do it right. Don't start the application,
start a script or batch file instead.

Post by Charles Wilson
Could we maybe extend the CYGWIN env var idea to files, similar to
the /etc/fstab[.d] structure, which are then *augmented* by $CYGWIN?

Reluctantly so. Opening files costs time. Reading the env is extremly
cheap in comparison.