Seeking developer to assist with adding OpenAFS Reparse Tag Support to Cygwin

[Jeffrey Altman]

The OpenAFS 1.7.x release series is implemented as a native Windows File
System Redirector. Microsoft has assigned

IO_REPARSE_TAG_OPENAFS_DFS 0x00000037L

which is used to identify three types of objects as reparse points.

1. AFS Mount Points. An AFS mount point is a directory entry which
refers to the root directory of another volume in the AFS name
space.

2. AFS Symlinks. An AFS Symlink behaves similar to a Posix symlink.
It can be either relative or absolute and can refer to any other
object in the file system. An object that is a symlink to a
directory will report FILE_ATTR_DIRECTORY and
FILE_ATTR_REPARSE_POINT. A symlink to any other object will report
just the FILE_ATTR_REPARSE_POINT flag.

Unlike a Posix symlink, an AFS symlink can include a component
whose tail is "@sys" which when evaluated will be replaced with
a string selected from an ordered list of "system names".

3. AFS UNC Redirection. This object refers to an arbitrary UNC path
that does not have to be in the AFS name space.

All reparse points are transparently handled by the file system.
However, as documented by Microsoft


http://msdn.microsoft.com/en-us/library/windows/desktop/aa365682%28v=vs.85%29.aspx

functions including:

GetFileAttributes
GetFileAttributesEx
GetFileSecurity
FindFirstFile
FindFirstFileEx
FindNextFile

will always return information about the reparse point object and not
the target. As Cygwin at the present time is unaware of the behavior of
the AFS reparse points, it discards them. However, this can result in
confusing behavior for applications and end users which are confused by
output that conflicts with that which would be obtained by

fh = CreateFile(path)
GetFileInformationByHandleW(fh, BasicInfo)
CloseHandle(hf)

I am seeking an experienced Cygwin developer to work with me to add
knowledge of AFS reparse points to the code base.

Thank you.

Jeffrey Altman
OpenAFS Gatekeeper

[Christopher Faylor]

Post by Jeffrey Altman
The OpenAFS 1.7.x release series is implemented as a native Windows File
System Redirector. Microsoft has assigned

Sorry but this mailing list isn't intended to be a place where you
advertise your needs. If you want to contribute to Cygwin that's great.
If you have questions about the code, ask away.

First please demonstrate that you've taken at least a modicum of time
to understand the code.

cgf

[Jeffrey Altman]

Christopher,

The request is for a developer to assist. I am more than capable of
providing code. As a contributor to open source projects for more than
25 years I am not going to make assumptions about how those who have
developed and maintained Cygwin want the software to behave.

In addition, although I've sent Red Hat a contributor letter I do
not intend to be an active contributor to the project on a long term
basis. I have enough responsibilities on my plate with OpenAFS, Heimdal
Kerberos and Network Identity Manager.

Sincerely,

Jeffrey Altman

P.S. - The anoncvs service access described at

http://cygwin.com/cvs.html

is currently not working.

Post by Christopher Faylor

Post by Jeffrey Altman
The OpenAFS 1.7.x release series is implemented as a native Windows File
System Redirector. Microsoft has assigned

Sorry but this mailing list isn't intended to be a place where you
advertise your needs. If you want to contribute to Cygwin that's great.
If you have questions about the code, ask away.
First please demonstrate that you've taken at least a modicum of time
to understand the code.
cgf

[marco atzeri]

Post by Jeffrey Altman
P.S. - The anoncvs service access described at
http://cygwin.com/cvs.html
is currently not working.

Hi Jeffrey,
just try again later, recently the server seems overloaded
and the service availability is a bit random

it worked 3 minutes ago, and not worked 1 minute ago

Regards
Marco

[Jeffrey Altman]

Post by marco atzeri
Hi Jeffrey,
just try again later, recently the server seems overloaded
and the service availability is a bit random
it worked 3 minutes ago, and not worked 1 minute ago
Regards
Marco

Thanks Marco.

[Christopher Faylor]

Post by Jeffrey Altman
The request is for a developer to assist. I am more than capable of
providing code. As a contributor to open source projects for more than
25 years I am not going to make assumptions about how those who have
developed and maintained Cygwin want the software to behave.

If you are looking for pointers on where to start, look at the
fhandler*.cc files and path.cc/path.h. If you have some suggestions
on how you'd like to add the functionality we can provide pointers.

And, although they are out-of-date, the how-*-works.txt files in
the cygwin directory may provide some information.

Post by Jeffrey Altman
P.S. - The anoncvs service access described at
http://cygwin.com/cvs.html
is currently not working.

I see that Marco has suggested that you were experiencing the anonymous
cvs restriction which kicks in when sourceware load average is too high.

If that isn't the case, you'll need to provide more details.

cgf

[Corinna Vinschen]

Post by Christopher Faylor

Post by Jeffrey Altman
The request is for a developer to assist. I am more than capable of
providing code. As a contributor to open source projects for more than
25 years I am not going to make assumptions about how those who have
developed and maintained Cygwin want the software to behave.

If you are looking for pointers on where to start, look at the
fhandler*.cc files and path.cc/path.h. If you have some suggestions
on how you'd like to add the functionality we can provide pointers.

The core routine is symlink_info::check_reparse_point in path.cc.

Actually, Cygwin always opens files with FILE_OPEN_REPARSE_POINT
and then, if it *is* a reparse point, it reads the contents of the
reparse point and expects it to be of type IO_REPARSE_TAG_SYMLINK
or IO_REPARSE_TAG_MOUNT_POINT. Those are handled as symlinks.
Other types are not recognized and the FILE_ATTRIBUTE_REPARSE_POINT
attribute is deleted from the DOS attributes. The later file open
call in fhandler.cc will omit the FILE_OPEN_REPARSE_POINT flag from
the open flags, so it will open the file transparently via the
reparse redirector.

But there's a chance that this doesn't happen in all circumstances as
expected. As a first cut, look for usage of FILE_OPEN_REPARSE_POINT and
the usage of the path_conv::is_rep_symlink method thoughout fhandler*.cc
and syscalls.cc.


HTH,
Corinna

[Jeffrey Altman]

Post by Corinna Vinschen
The core routine is symlink_info::check_reparse_point in path.cc.
Actually, Cygwin always opens files with FILE_OPEN_REPARSE_POINT
and then, if it *is* a reparse point, it reads the contents of the
reparse point and expects it to be of type IO_REPARSE_TAG_SYMLINK
or IO_REPARSE_TAG_MOUNT_POINT. Those are handled as symlinks.
Other types are not recognized and the FILE_ATTRIBUTE_REPARSE_POINT
attribute is deleted from the DOS attributes. The later file open
call in fhandler.cc will omit the FILE_OPEN_REPARSE_POINT flag from
the open flags, so it will open the file transparently via the
reparse redirector.
But there's a chance that this doesn't happen in all circumstances as
expected. As a first cut, look for usage of FILE_OPEN_REPARSE_POINT and
the usage of the path_conv::is_rep_symlink method thoughout fhandler*.cc
and syscalls.cc.

Thank you Corinna.

In my initial e-mail I pointed to an MSDN document which describes the
inconsistent behavior of the Win32 API with regards to reparse points.
The key thing to take away from that is that regardless of what the
application (in this case Cygwin) wants, FILE_OPEN_REPARSE_POINT will be
included in the underlying CreateFile() call for operations such
as

GetFileAttributes[Ex]
GetFileSecurity
Directory enumeration

I believe the rational is that these functions should be fast and should
not block if the reparse point target is located "far away"
or perhaps "offline" in a hierarchical storage management system.

One side effect of this for Cygwin is that reported sizes of objects in
the "ls -l" output is incorrect. For example, ti.exe is a symlink to a
file stored in AFS.

11/01/2012 0:27 <SYMLINK> ti.exe
[\afs\your-file-system.com\public\jaltman\TI_TAV_5.0_EN_Full.exe]

The actual size of the target object is:

3/26/2012 20:32 103,103,720 TI_TAV_5.0_EN_Full.exe

but Cygwin lists it as:

-rwxr-xr-x 1 jaltman None 63 Nov 1 01:27 ti.exe

where 63 bytes is the size of the reparse point data not the target. If
Cygwin is going to discard FILE_ATTRIBUTE_REPARSE_POINT, then it must
also resolve the target attributes when collecting stat info.

This can be done by opening the file with CreateFile(path, no
FILE_OPEN_REPARSE_POINT) and then using GetFileInformationByHandle().

http://msdn.microsoft.com/en-us/library/windows/desktop/aa364952%28v=vs.85%29.aspx

I don't know if this is a behavior that the Cygwin community wants to
alter. It is certainly confusing to end users and potentially breaks
applications which believe a file is smaller than it is. However,
resolving reparse targets can be very expensive in some cases.

Sincerely,

Jeffrey Altman

[Corinna Vinschen]

Post by Jeffrey Altman

Post by Corinna Vinschen
The core routine is symlink_info::check_reparse_point in path.cc.
Actually, Cygwin always opens files with FILE_OPEN_REPARSE_POINT
and then, if it *is* a reparse point, it reads the contents of the
reparse point and expects it to be of type IO_REPARSE_TAG_SYMLINK
or IO_REPARSE_TAG_MOUNT_POINT. Those are handled as symlinks.
Other types are not recognized and the FILE_ATTRIBUTE_REPARSE_POINT
attribute is deleted from the DOS attributes. The later file open
call in fhandler.cc will omit the FILE_OPEN_REPARSE_POINT flag from
the open flags, so it will open the file transparently via the
reparse redirector.
But there's a chance that this doesn't happen in all circumstances as
expected. As a first cut, look for usage of FILE_OPEN_REPARSE_POINT and
the usage of the path_conv::is_rep_symlink method thoughout fhandler*.cc
and syscalls.cc.

Thank you Corinna.
In my initial e-mail I pointed to an MSDN document which describes the
inconsistent behavior of the Win32 API with regards to reparse points.
The key thing to take away from that is that regardless of what the
application (in this case Cygwin) wants, FILE_OPEN_REPARSE_POINT will be
included in the underlying CreateFile() call for operations such
as
GetFileAttributes[Ex]
GetFileSecurity
Directory enumeration

Did you look into the code? Cygwin doesn't use WIn32 functions to
access the file system. It uses the underlying native NT functions.

Post by Jeffrey Altman
-rwxr-xr-x 1 jaltman None 63 Nov 1 01:27 ti.exe
where 63 bytes is the size of the reparse point data not the target. If
Cygwin is going to discard FILE_ATTRIBUTE_REPARSE_POINT, then it must
also resolve the target attributes when collecting stat info.

http://cygwin.com/acronyms/#PTC


Corinna

[Corinna Vinschen]

Post by Corinna Vinschen

Post by Jeffrey Altman

Post by Corinna Vinschen
The core routine is symlink_info::check_reparse_point in path.cc.
Actually, Cygwin always opens files with FILE_OPEN_REPARSE_POINT
and then, if it *is* a reparse point, it reads the contents of the
reparse point and expects it to be of type IO_REPARSE_TAG_SYMLINK
or IO_REPARSE_TAG_MOUNT_POINT. Those are handled as symlinks.
Other types are not recognized and the FILE_ATTRIBUTE_REPARSE_POINT
attribute is deleted from the DOS attributes. The later file open
call in fhandler.cc will omit the FILE_OPEN_REPARSE_POINT flag from
the open flags, so it will open the file transparently via the
reparse redirector.
But there's a chance that this doesn't happen in all circumstances as
expected. As a first cut, look for usage of FILE_OPEN_REPARSE_POINT and
the usage of the path_conv::is_rep_symlink method thoughout fhandler*.cc
and syscalls.cc.

Thank you Corinna.
In my initial e-mail I pointed to an MSDN document which describes the
inconsistent behavior of the Win32 API with regards to reparse points.
The key thing to take away from that is that regardless of what the
application (in this case Cygwin) wants, FILE_OPEN_REPARSE_POINT will be
included in the underlying CreateFile() call for operations such
as
GetFileAttributes[Ex]
GetFileSecurity
Directory enumeration

Did you look into the code? Cygwin doesn't use WIn32 functions to
access the file system. It uses the underlying native NT functions.

Post by Jeffrey Altman
-rwxr-xr-x 1 jaltman None 63 Nov 1 01:27 ti.exe
where 63 bytes is the size of the reparse point data not the target. If
Cygwin is going to discard FILE_ATTRIBUTE_REPARSE_POINT, then it must
also resolve the target attributes when collecting stat info.

http://cygwin.com/acronyms/#PTC

Btw. The right thing to do here is not to tweak stat or any other API
call. The right thing to do is to fix symlink_info::check_reparse_point
to recognize the AFS reparse point and treat it as a symlink, just like
the IO_REPARSE_TAG_SYMLINK and IO_REPARSE_TAG_MOUNT_POINT types.
The returned path should be a Cygwin-compatible POSIX path pointing to
the real file. In your above example, Cygwin's ls should handle
ti.exe as a symlink to the AFS file.


Corinna

[Jeffrey Altman]

Post by Corinna Vinschen
Btw. The right thing to do here is not to tweak stat or any other API
call. The right thing to do is to fix symlink_info::check_reparse_point
to recognize the AFS reparse point and treat it as a symlink, just like
the IO_REPARSE_TAG_SYMLINK and IO_REPARSE_TAG_MOUNT_POINT types.
The returned path should be a Cygwin-compatible POSIX path pointing to
the real file. In your above example, Cygwin's ls should handle
ti.exe as a symlink to the AFS file.

Corinna,

There are two issues here.

1. Adding support for AFS Reparse Tag Data.

2. Fixing Cygwin to do the right thing for other file systems that
Cygwin does not recognize.

I already have rough code that adds the AFS Reparse Tag Data. Although
it has neither been compiled nor tested.

I haven't looked at the second issue. I have simply been describing it
as the motivation for wanting to do the first.

Jeffrey Altman

[Jeffrey Altman]

Post by Corinna Vinschen
Did you look into the code? Cygwin doesn't use WIn32 functions to
access the file system. It uses the underlying native NT functions.

As you know, Win32 functions are thin wrappers on the Nt functions. The
behavior exhibited by Cygwin is the same. Either Cygwin is replicating
the behavior of the Win32 functions or the choice of always accessing
the reparse point information is provided by the Nt functions.

For someone with a working build environment and a debugger it would be
fairly simple to step through the code. I'm not yet at the point of
having a working Cygwin build environment.

While I am not 100% sure, I suspect the issue is that the attribute
information returned from NtQueryDirectoryFile is used (minus the
removal of the FILE_ATTRIBUTE_REPARSE_POINT flag). What needs to happen
is that when the FILE_ATTRIBUTE_REPARSE_POINT flag is removed the target
object must be opened and the FILE_STANDARD_INFO and FILE_INTERNAL_INFO
must be obtained.

Jeffrey Altman

[Jeffrey Altman]

Post by Christopher Faylor
I see that Marco has suggested that you were experiencing the anonymous
cvs restriction which kicks in when sourceware load average is too high.

Christopher,

My issue was the anoncvs restriction. After dozens of repeated attempts
I was able to checkout the repository. It would be helpful if the
limitation was documented in the FAQ or on the cvs.html page.

Thank you.

Jeffrey Altman

[Corinna Vinschen]

Hi Jeffrey,

Post by Jeffrey Altman
Christopher,
The request is for a developer to assist. I am more than capable of
providing code. As a contributor to open source projects for more than
25 years I am not going to make assumptions about how those who have
developed and maintained Cygwin want the software to behave.
In addition, although I've sent Red Hat a contributor letter [...]

Your CA arrived and is signed.


Thanks,
Corinna

[Jeffrey Altman]

Post by Corinna Vinschen
Your CA arrived and is signed.
Thanks,
Corinna

Thanks Corinna,

I have built the cvs head as of this afternoon. "make check" in
build/i686-pc-cygwin/winsup/ cannot complete:

make[1]: Entering directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/cygwin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/cygwin'
make[1]: Entering directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/testsuite'
make[1]: *** No rule to make target `dataascii.o', needed by `libltp.a'.
Stop.
make[1]: Leaving directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/testsuite'
Makefile:109: recipe for target `check' failed
make: *** [check] Error 2


However, I have tested the build product with the attached patch. Feel
free to adjust the formatting, comments, etc.

Do not hesitate to ask if you have any questions.

Jeffrey Altman

[Christopher Faylor]

Post by Jeffrey Altman

Post by Corinna Vinschen
Your CA arrived and is signed.
Thanks,
Corinna

Thanks Corinna,
I have built the cvs head as of this afternoon. "make check" in
make[1]: Entering directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/cygwin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/cygwin'
make[1]: Entering directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/testsuite'
make[1]: *** No rule to make target `dataascii.o', needed by `libltp.a'.
Stop.
make[1]: Leaving directory
`/cygdrive/d/src/cygwin.cvs/build/i686-pc-cygwin/winsup/testsuite'
Makefile:109: recipe for target `check' failed
make: *** [check] Error 2
However, I have tested the build product with the attached patch. Feel
free to adjust the formatting, comments, etc.
Do not hesitate to ask if you have any questions.

Could you reformat that without the MS-DOS line endings.

cgf

[Jeffrey Altman]

Post by Christopher Faylor

Post by Jeffrey Altman
Do not hesitate to ask if you have any questions.

Could you reformat that without the MS-DOS line endings.
cgf

Here is the same file after running 'dos2unix' on it.

[Christopher Faylor]

Post by Jeffrey Altman

Post by Christopher Faylor

Post by Jeffrey Altman
Do not hesitate to ask if you have any questions.

Could you reformat that without the MS-DOS line endings.

Here is the same file after running 'dos2unix' on it.

Sorry to nickel and dime you, but convention is to use unified diff
format when submitting patches. I should have mentioned that before.

Post by Jeffrey Altman
From what I can see, your changes don't follow the indentation

convention of the code around it, i.e., the changes in path.h
seem to use four spaces for indentation.

cgf

[Corinna Vinschen]

Post by Christopher Faylor

Post by Jeffrey Altman

Post by Christopher Faylor

Post by Jeffrey Altman
Do not hesitate to ask if you have any questions.

Could you reformat that without the MS-DOS line endings.

Here is the same file after running 'dos2unix' on it.

Sorry to nickel and dime you, but convention is to use unified diff
format when submitting patches. I should have mentioned that before.

Post by Jeffrey Altman
From what I can see, your changes don't follow the indentation

convention of the code around it, i.e., the changes in path.h
seem to use four spaces for indentation.

A ChangeLog entry would be helpful, too.

This is all nicely described on http://cygwin.com/contrib.html
under the "When you have finalized your changes" section.


Thanks,
Corinna

[Jeffrey Altman]

Post by Corinna Vinschen

Post by Christopher Faylor

Post by Jeffrey Altman

Post by Christopher Faylor

Post by Jeffrey Altman
Do not hesitate to ask if you have any questions.

Could you reformat that without the MS-DOS line endings.

Here is the same file after running 'dos2unix' on it.

Sorry to nickel and dime you, but convention is to use unified diff
format when submitting patches. I should have mentioned that before.

Post by Jeffrey Altman
From what I can see, your changes don't follow the indentation

convention of the code around it, i.e., the changes in path.h
seem to use four spaces for indentation.

A ChangeLog entry would be helpful, too.

Please consider converting to 'git' and deploying 'gerrit' for code
review. You will find it makes not only your only life happier but make
it make easier on flyby contributors.

Post by Corinna Vinschen
This is all nicely described on http://cygwin.com/contrib.html
under the "When you have finalized your changes" section.

I will read.