Discussion:
New Cygwin family member
Corinna Vinschen
2013-02-05 16:11:51 UTC
Permalink
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Teemu Nätkinniemi
2013-02-05 17:14:13 UTC
Permalink
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Congratulations Corinna!

Any idea when snapshots and toolchain are available for eager early
adopters?

This reminds of the time when I got the first look of b19.

Teemu
Corinna Vinschen
2013-02-05 17:57:28 UTC
Permalink
Post by Teemu Nätkinniemi
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Congratulations Corinna!
Any idea when snapshots and toolchain are available for eager early
adopters?
Not at all. For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.

Right now I'm just building simple test applications and there's still
important stuff broken, probably even in the toolchain.


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Chris Sutcliffe
2013-02-06 02:04:11 UTC
Permalink
Post by Corinna Vinschen
Post by Teemu Nätkinniemi
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Congratulations Corinna!
Any idea when snapshots and toolchain are available for eager early
adopters?
Not at all. For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.
For us poor schmucks stuck in the 32-bit world, I'm hoping there will
eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).

Cheers,

Chris

--
Chris Sutcliffe
http://emergedesktop.org
http://www.google.com/profiles/ir0nh34d
Yaakov (Cygwin/X)
2013-02-06 04:14:30 UTC
Permalink
Post by Chris Sutcliffe
Post by Corinna Vinschen
Not at all. For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.
For us poor schmucks stuck in the 32-bit world, I'm hoping there will
eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?
That will be possible as soon as the patches are released.
Post by Chris Sutcliffe
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).
The problem is that not *everything* can be cross-compiled easily, if
at all. Linux distributions generally use dedicated build
machines/farms (e.g. koji for Fedora) for this very reason. This will
certainly need to be addressed when the time comes.


Yaakov
Corinna Vinschen
2013-02-06 11:20:20 UTC
Permalink
Post by Yaakov (Cygwin/X)
Post by Chris Sutcliffe
Post by Corinna Vinschen
Not at all. For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.
For us poor schmucks stuck in the 32-bit world, I'm hoping there will
eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?
That will be possible as soon as the patches are released.
The patches will be relative to current binutils CVS / gcc SVN. I'd
love to publish the patches, but just yesterday we encountered a big
problem in terms of the auto-import mechanism which I really hope to fix
before.

Without going into too much detail, the problem is that gcc references
external variables using pc-relative addressing with 32 bit displacement.
Only if a symbol is marked as dllimport, gcc will instead generate
indirect addressing opcodes. So, if a source uses some external
variable from a DLL, but neglects to declare it as dllimport symbol,
the following will happen:

extern char *optarg;
char *o = optarg;

This code will be translated into something along these lines:

movq optarg(%rip), %rax

The "optarg" here is a signed 32 bit displacement. However, if the
offset between executable and the exporting DLL (in this case the cygwin
DLL) is bigger than 32 bit, which is *pretty* likely, given that we
want to place our DLLs where Windows will not collide with us.

The end result is either a SEGV, if the resulting address is outside of
readable virtual mem, or an access to the wrong data.

The easy solution is to stick to the current methods and play the
"serves them right" card if they didn't declare the variable as
dllimport. However, from the POSIX POV the application didn't do
anything wrong. And apart from the fact that a SEGV in this seemingly
innocent situation is pretty mind-boggling, it's also quite annoying to
have working 32 bit code but crashing 64 bit code generated from the
same source.

That's why I hope we can come up with a better solution.

If anybody is interested in the patches in the current state, I can
send them to the list anyway, though.
Post by Yaakov (Cygwin/X)
Post by Chris Sutcliffe
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).
The problem is that not *everything* can be cross-compiled easily, if
at all. Linux distributions generally use dedicated build
machines/farms (e.g. koji for Fedora) for this very reason. This will
certainly need to be addressed when the time comes.
Yeah. This will still be a heck of a lot of work. Given that "Hello
World" is barely a day old, you may understand that I didn't even dare
to call fork yet :}

Another really big problem is entirely unsolved yet: 32 and 64 bit
Cygwin processes execing each other, as well as working together.


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Corinna Vinschen
2013-02-08 13:55:08 UTC
Permalink
Post by Corinna Vinschen
Post by Yaakov (Cygwin/X)
Post by Chris Sutcliffe
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).
The problem is that not *everything* can be cross-compiled easily, if
at all. Linux distributions generally use dedicated build
machines/farms (e.g. koji for Fedora) for this very reason. This will
certainly need to be addressed when the time comes.
Yeah. This will still be a heck of a lot of work. Given that "Hello
World" is barely a day old, you may understand that I didn't even dare
to call fork yet :}
Yay! I have a running dash and fork works! <at this point, imagine
the *plop* sound of opening a bottle of finest champagne>
Post by Corinna Vinschen
Another really big problem is entirely unsolved yet: 32 and 64 bit
Cygwin processes execing each other, as well as working together.
Another problem is the memory layout (the "fork" problem). Fortunately
we have a 43 bit address space now. Here's is our current model, still
open for discussion:

- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
0x700:00000000 up to 0x7ff:ffffffff since these areas are used
by the OS for ... everything.

- Cygwin thread stacks will be located between 0x0:80000000 and
0x0:ffffffff. With a default stack size of 1 Meg, we have room for
2048 threads.

- The Cygwin executables will be loaded to 0x1:00040000.

- The Cygwin DLL will be loaded to 0x1:80040000. That leaves 2 Gigs of
space for the executable. The space from 0x1:8000000 up to
0x1:80040000 will be used for Cygwin's shared memory areas.

- Other Cygwin distro DLLs are supposed to be rebased to the area
between 0x2:00000000 up to 0x3:ffffffff. This is an area of 8 Gigs
for DLLs. That should be enough for a while, I guess.

- The heap will be located at 0x4:00000000, and it will be 512Megs by
default.

- Then ... a big void ...

- Eventually, mmap's will be allocated from 0x700:00000000 downwards.


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Corinna Vinschen
2013-02-08 14:10:24 UTC
Permalink
Post by Corinna Vinschen
Post by Corinna Vinschen
Post by Yaakov (Cygwin/X)
Post by Chris Sutcliffe
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).
The problem is that not *everything* can be cross-compiled easily, if
at all. Linux distributions generally use dedicated build
machines/farms (e.g. koji for Fedora) for this very reason. This will
certainly need to be addressed when the time comes.
Yeah. This will still be a heck of a lot of work. Given that "Hello
World" is barely a day old, you may understand that I didn't even dare
to call fork yet :}
Yay! I have a running dash and fork works! <at this point, imagine
the *plop* sound of opening a bottle of finest champagne>
Post by Corinna Vinschen
Another really big problem is entirely unsolved yet: 32 and 64 bit
Cygwin processes execing each other, as well as working together.
Another problem is the memory layout (the "fork" problem). Fortunately
we have a 43 bit address space now. Here's is our current model, still
- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
0x700:00000000 up to 0x7ff:ffffffff since these areas are used
by the OS for ... everything.
- Cygwin thread stacks will be located between 0x0:80000000 and
0x0:ffffffff. With a default stack size of 1 Meg, we have room for
2048 threads.
- The Cygwin executables will be loaded to 0x1:00040000.
Correction: Not 0x1:0004.0000 but 0x1:0040.0000 Exactly the old
default start address for executables + 4 Gigs.
Post by Corinna Vinschen
- The Cygwin DLL will be loaded to 0x1:80040000. That leaves 2 Gigs of
space for the executable. The space from 0x1:8000000 up to
0x1:80040000 will be used for Cygwin's shared memory areas.
- Other Cygwin distro DLLs are supposed to be rebased to the area
between 0x2:00000000 up to 0x3:ffffffff. This is an area of 8 Gigs
for DLLs. That should be enough for a while, I guess.
- The heap will be located at 0x4:00000000, and it will be 512Megs by
default.
- Then ... a big void ...
- Eventually, mmap's will be allocated from 0x700:00000000 downwards.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Corinna Vinschen
2013-02-12 11:00:36 UTC
Permalink
Hi guys,
Post by Corinna Vinschen
Another problem is the memory layout (the "fork" problem). Fortunately
we have a 43 bit address space now. Here's is our current model, still
- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
0x700:00000000 up to 0x7ff:ffffffff since these areas are used
by the OS for ... everything.
- Cygwin thread stacks will be located between 0x0:80000000 and
0x0:ffffffff. With a default stack size of 1 Meg, we have room for
2048 threads.
- The Cygwin executables will be loaded to 0x1:00400000.
- The Cygwin DLL will be loaded to 0x1:80040000. That leaves 2 Gigs of
space for the executable. The space from 0x1:8000000 up to
0x1:80040000 will be used for Cygwin's shared memory areas.
A slight change to the memory model occured to me, which should drop
Post by Corinna Vinschen
- Other Cygwin distro DLLs are supposed to be rebased to the area
between 0x2:00000000 up to 0x3:ffffffff. This is an area of 8 Gigs
for DLLs. That should be enough for a while, I guess.
The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
used by rebase{all}...

...while ld's --auto-image-base will use the region from 0x4:00000000 up
to 0x6:00000000.

This means, rebased DLLs will never collide with auto-image-based DLLs,
and both types of DLLs have a full 8 Gigs space each, which should be
enough for the forseeable future.
Post by Corinna Vinschen
- The heap will be located at 0x4:00000000, and it will be 512Megs by
default.
The heap will then be located at 0x6:00000000, of course.
Post by Corinna Vinschen
- Then ... a big void ...
- Eventually, mmap's will be allocated from 0x700:00000000 downwards.
This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
about 7 Terabytes. I hope that's ok, even for our friends on the
Fortran frontier.

Comments?


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Ryan Johnson
2013-02-12 13:37:38 UTC
Permalink
Post by Corinna Vinschen
Hi guys,
Post by Corinna Vinschen
Another problem is the memory layout (the "fork" problem). Fortunately
we have a 43 bit address space now. Here's is our current model, still
- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
0x700:00000000 up to 0x7ff:ffffffff since these areas are used
by the OS for ... everything.
- Cygwin thread stacks will be located between 0x0:80000000 and
0x0:ffffffff. With a default stack size of 1 Meg, we have room for
2048 threads.
- The Cygwin executables will be loaded to 0x1:00400000.
- The Cygwin DLL will be loaded to 0x1:80040000. That leaves 2 Gigs of
space for the executable. The space from 0x1:8000000 up to
0x1:80040000 will be used for Cygwin's shared memory areas.
A slight change to the memory model occured to me, which should drop
Post by Corinna Vinschen
- Other Cygwin distro DLLs are supposed to be rebased to the area
between 0x2:00000000 up to 0x3:ffffffff. This is an area of 8 Gigs
for DLLs. That should be enough for a while, I guess.
The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
used by rebase{all}...
...while ld's --auto-image-base will use the region from 0x4:00000000 up
to 0x6:00000000.
This means, rebased DLLs will never collide with auto-image-based DLLs,
and both types of DLLs have a full 8 Gigs space each, which should be
enough for the forseeable future.
Post by Corinna Vinschen
- The heap will be located at 0x4:00000000, and it will be 512Megs by
default.
The heap will then be located at 0x6:00000000, of course.
Post by Corinna Vinschen
- Then ... a big void ...
- Eventually, mmap's will be allocated from 0x700:00000000 downwards.
This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
about 7 Terabytes. I hope that's ok, even for our friends on the
Fortran frontier.
Comments?
If I count correctly:
- Total address space available is 8TB
- Windows no-man's land regions occupy 2GB and 1TB, respectively
- Cygwin thread stacks occupy 2GB
- Executable image gets 2GB
- distro and auto-image dlls together occupy 16GB (including some tens
of MB for the cygwin dll and shared memory areas)
- heap/mmap area gets from 0x6: to 0x700:, which I compute as 7144GB

In other words, the proposed layout occupies only 20GB, or less than
0.3% of the address space. If somebody wants significantly more heap
space, they'd have to brave the 1TB no-man's land and risk fork
failures. Or, more likely, just wait for Intel and MS to bump the
address space limit by a few powers of two (which should happen roughly
when the "normal" amount of memory to have in a Windows box approaches
8TB). At that point, the heap/mmap area can easily take over the
additional space.

What's not to like?
Ryan
Corinna Vinschen
2013-02-12 13:57:46 UTC
Permalink
Post by Ryan Johnson
Post by Corinna Vinschen
Hi guys,
Post by Corinna Vinschen
Another problem is the memory layout (the "fork" problem). Fortunately
we have a 43 bit address space now. Here's is our current model, still
- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
0x700:00000000 up to 0x7ff:ffffffff since these areas are used
by the OS for ... everything.
- Cygwin thread stacks will be located between 0x0:80000000 and
0x0:ffffffff. With a default stack size of 1 Meg, we have room for
2048 threads.
- The Cygwin executables will be loaded to 0x1:00400000.
- The Cygwin DLL will be loaded to 0x1:80040000. That leaves 2 Gigs of
space for the executable. The space from 0x1:8000000 up to
0x1:80040000 will be used for Cygwin's shared memory areas.
A slight change to the memory model occured to me, which should drop
Post by Corinna Vinschen
- Other Cygwin distro DLLs are supposed to be rebased to the area
between 0x2:00000000 up to 0x3:ffffffff. This is an area of 8 Gigs
for DLLs. That should be enough for a while, I guess.
The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
used by rebase{all}...
...while ld's --auto-image-base will use the region from 0x4:00000000 up
to 0x6:00000000.
This means, rebased DLLs will never collide with auto-image-based DLLs,
and both types of DLLs have a full 8 Gigs space each, which should be
enough for the forseeable future.
Post by Corinna Vinschen
- The heap will be located at 0x4:00000000, and it will be 512Megs by
default.
The heap will then be located at 0x6:00000000, of course.
Post by Corinna Vinschen
- Then ... a big void ...
- Eventually, mmap's will be allocated from 0x700:00000000 downwards.
This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
about 7 Terabytes. I hope that's ok, even for our friends on the
Fortran frontier.
Comments?
- Total address space available is 8TB
- Windows no-man's land regions occupy 2GB and 1TB, respectively
- Cygwin thread stacks occupy 2GB
- Executable image gets 2GB
- distro and auto-image dlls together occupy 16GB (including some
tens of MB for the cygwin dll and shared memory areas)
Not quite :) You missed the fact that the Cygwin DLL is located
outside that region, at 0x1:80000000. This includes Cygwin's
shared memory and allows a cygheap size of about 2 GiB. Still,
that doesn't reduce the heap/mmap size at all, so there's still
7144 GB left.
Post by Ryan Johnson
- heap/mmap area gets from 0x6: to 0x700:, which I compute as 7144GB
In other words, the proposed layout occupies only 20GB, or less than
0.3% of the address space. If somebody wants significantly more heap
space, they'd have to brave the 1TB no-man's land and risk fork
failures. Or, more likely, just wait for Intel and MS to bump the
address space limit by a few powers of two (which should happen
roughly when the "normal" amount of memory to have in a Windows box
approaches 8TB). At that point, the heap/mmap area can easily take
over the additional space.
What's not to like?
Ryan
Yeah, that doesn't sound overly cramped, does it?


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Yaakov (Cygwin/X)
2013-02-06 02:06:01 UTC
Permalink
Post by Corinna Vinschen
Not at all. For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.
Congratulations, and nice work! Please let me know if I can assist with
the new toolchains.


Yaakov
Daniel Colascione
2013-02-06 02:06:51 UTC
Permalink
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Awesome! Is it too early to ask about performance? I'm interested in whether
Cygwin's running under WOW64 incurs much of a performance hit, and before now,
an apples-to-apples comparison with native process execution hasn't been possible.
Corinna Vinschen
2013-02-06 11:20:43 UTC
Permalink
Post by Daniel Colascione
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Awesome! Is it too early to ask about performance?
Yes, very definitely!


Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Maintainer cygwin AT cygwin DOT com
Red Hat
Larry Hall (Cygwin Developers)
2013-02-06 03:05:24 UTC
Permalink
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
And it must have all 64-digits! Nice. :-)
--
Larry
Herbert Stocker
2013-02-06 03:12:38 UTC
Permalink
Welcome from Munich, to this new citizen of the computing world.
Does it already have a name? Like "cyg64.exe".

Actually i really appreciate this work, Corinna.
Like i do appreciate the whole Cygwin project. A computer would not be
complete
if it wouldn't have a command line interface with a full-featured
command language
like bash provides.

Big Appreciation to all the programmers, testers and writers of the
Cygwin project
from me!

Herbert
Post by Corinna Vinschen
Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day. The process is alive and well. cyg64win1.dll, the proud mother,
is doing fine under the circumstances.
Corinna
Loading...