diff --git a/copyfile.3 b/copyfile.3 index 55d4454..08d07a2 100644 --- a/copyfile.3 +++ b/copyfile.3 @@ -125,13 +125,15 @@ file had extended attributes but no ACLs, the return value would be .It Dv COPYFILE_PACK Serialize the .Va from -file. The +file. +The .Va to file is an AppleDouble-format file. .It Dv COPYFILE_UNPACK Unserialize the .Va from -file. The +file. +The .Va from file is an AppleDouble-format file; the .Va to @@ -168,7 +170,8 @@ file. (This is only applicable for the .Fn copyfile function.) No error is returned if .Xr remove 3 -fails. Note that +fails. +Note that .Xr remove 3 removes a symbolic link itself, not the target of the link. @@ -185,8 +188,8 @@ This flag is equivalent to (COPYFILE_EXCL | COPYFILE_ACL | COPYFILE_STAT | COPYF | COPYFILE_NOFOLLOW_SRC). Note that if cloning is successful, progress callbacks will not be invoked. Note also that there is no support for cloning directories: if a directory is provided as the source, -an error will be returned. Since this flag implies COPYFILE_NOFOLLOW_SRC, symbolic links themselves will -be cloned instead of their targets. +an error will be returned. +Since this flag implies COPYFILE_NOFOLLOW_SRC, symbolic links themselves will be cloned instead of their targets. (This is only applicable for the .Fn copyfile function.) @@ -197,9 +200,9 @@ This flag is equivalent to (COPYFILE_EXCL | COPYFILE_ACL | COPYFILE_STAT | COPYF | COPYFILE_NOFOLLOW_SRC). Note that if cloning is successful, progress callbacks will not be invoked. Note also that there is no support for cloning directories: if a directory is provided as the source and -COPYFILE_CLONE_FORCE is not passed, this will instead copy the directory. Since this flag implies COPYFILE_NOFOLLOW_SRC, -symbolic links themselves will be cloned instead of their targets. Recursive copying however is -supported, see below for more information. +COPYFILE_CLONE_FORCE is not passed, this will instead copy the directory. +Since this flag implies COPYFILE_NOFOLLOW_SRC, symbolic links themselves will be cloned instead of their targets. +Recursive copying however is supported, see below for more information. (This is only applicable for the .Fn copyfile function.) @@ -216,16 +219,19 @@ if sparse copying cannot be performed for any reason; otherwise, an error is ret This is a convenience macro, equivalent to .Dv (COPYFILE_NOFOLLOW_DST | COPYFILE_NOFOLLOW_SRC) . .It Dv COPYFILE_RUN_IN_PLACE -If the src file has quarantine information, add the QTN_FLAG_DO_NOT_TRANSLOCATE flag to the quarantine information of the dst file. This allows a bundle to run in place instead of being translocated. +If the src file has quarantine information, add the QTN_FLAG_DO_NOT_TRANSLOCATE flag to the quarantine information of the dst file. +This allows a bundle to run in place instead of being translocated. .It Dv COPYFILE_PRESERVE_DST_TRACKED Preserve the UF_TRACKED flag at .Va to when copying metadata, regardless of whether .Va from -has it set. This flag is used in conjunction with COPYFILE_STAT, or COPYFILE_CLONE (for its fallback case). +has it set. +This flag is used in conjunction with COPYFILE_STAT, or COPYFILE_CLONE (for its fallback case). .El .Pp -Copying files into a directory is supported. If +Copying files into a directory is supported. +If .Va to is a directory, .Va from @@ -269,7 +275,8 @@ parameters are pointers to .It Dv COPYFILE_STATE_SRC_FILENAME .It Dv COPYFILE_STATE_DST_FILENAME Get or set the filename associated with the source (or destination) -file. If it has not been initialized yet, the value will be +file. +If it has not been initialized yet, the value will be .Dv NULL . For .Fn copyfile_state_set , @@ -385,7 +392,8 @@ entered. (That is, none of the filesystem objects contained within the directory have been copied yet.) .It Dv COPYFILE_RECURSE_DIR_CLEANUP The object being copied is a directory, and all of the -objects contained have been copied. At this stage, the destination directory +objects contained have been copied. +At this stage, the destination directory being copied will have any extra permissions that were added to allow the copying will be removed. .It Dv COPYFILE_RECURSE_ERROR @@ -403,15 +411,15 @@ The second argument to the call-back function will indicate the stage of the copy, and will be one of the following values: .Bl -tag -width COPYFILE_FINISH .It Dv COPYFILE_START -Before copying has begun. The third -parameter will be a newly-created +Before copying has begun. +The third parameter will be a newly-created .Vt copyfile_state_t object with the call-back function and context pre-loaded. .It Dv COPYFILE_FINISH After copying has successfully finished. .It Dv COPYFILE_ERR -Indicates an error has happened at some stage. If the -first argument to the call-back function is +Indicates an error has happened at some stage. +If the first argument to the call-back function is .Dv COPYFILE_RECURSE_ERROR , then an error occurred while processing the source hierarchy; otherwise, it will indicate what type of object was being copied, @@ -437,13 +445,14 @@ values: The copy will continue as expected. .It Dv COPYFILE_SKIP This object will be skipped, and the next object will -be processed. (Note that, when entering a directory. -returning +be processed. +(Note that, when entering a directory, returning .Dv COPYFILE_SKIP from the call-back function will prevent the contents of the directory from being copied.) .It Dv COPYFILE_QUIT -The entire copy is aborted at this stage. Any filesystem +The entire copy is aborted at this stage. +Any filesystem objects created up to this point will remain. .Fn copyfile will return -1, but @@ -474,11 +483,13 @@ Note that recursive cloning is also supported with the .Dv COPYFILE_CLONE flag (but not the .Dv COPYFILE_CLONE_FORCE -flag). A recursive clone operation invokes +flag). +A recursive clone operation invokes .Fn copyfile with .Dv COPYFILE_CLONE -on every entry found in the source file-system object. Because +on every entry found in the source file-system object. +Because .Fn copyfile does not allow the cloning of directories, a recursive clone will instead copy any directory it finds (while cloning its contents). @@ -503,10 +514,12 @@ Note that if the source path ends in a .Va / its contents are copied rather than the directory itself (like cp(1)). The behavior of a recursive copy on a directory hierarchy also depends -on the contents of the destination. If the destination is a directory, -the source directory (or its contents, if the source path ends in a +on the contents of the destination. +If the destination is a directory, the source directory (or its contents, +if the source path ends in a .Va / -) will be copied into it. If the destination exists but is not a +) will be copied into it. +If the destination exists but is not a directory, and the source is a non-empty directory, the copy will fail; the exact error set depends on the flags provided to .Fn copyfile @@ -518,9 +531,11 @@ and .Fn fcopyfile will also use a callback to report data (e.g., .Dv COPYFILE_DATA ) -progress. If given, the callback will be invoked on each +progress. +If given, the callback will be invoked on each .Xr write 2 -call. The first argument to the callback function will be +call. +The first argument to the callback function will be .Dv COPYFILE_COPY_DATA . The second argument will either be .Dv COPYFILE_PROGRESS @@ -560,7 +575,8 @@ when finished with each individual attribute. may be called for all of the extended attributes, before the first callback with .Dv COPYFILE_PROGRESS -is invoked.) Any attribute skipped by returning +is invoked.) +Any attribute skipped by returning .Dv COPYFILE_SKIP from the .Dv COPYFILE_START @@ -621,6 +637,28 @@ changes while the copy is occurring, the results are undefined. does not reset the seek position for either source or destination. This can result in the destination file being a different size than the source file. +.Sh EXAMPLES +.Bd -literal -offset indent +/* Initialize a state variable */ +copyfile_state_t s; +s = copyfile_state_alloc(); +/* Copy the data and extended attributes of one file to another */ +copyfile("/tmp/f1", "/tmp/f2", s, COPYFILE_DATA | COPYFILE_XATTR); +/* Convert a file to an AppleDouble file for serialization */ +copyfile("/tmp/f2", "/tmp/tmpfile", NULL, COPYFILE_ALL | COPYFILE_PACK); +/* Release the state variable */ +copyfile_state_free(s); +/* A more complex way to call copyfile() */ +s = copyfile_state_alloc(); +copyfile_state_set(s, COPYFILE_STATE_SRC_FILENAME, "/tmp/foo"); +/* One of src or dst must be set... rest can come from the state */ +copyfile(NULL, "/tmp/bar", s, COPYFILE_ALL); +/* Now copy the same source file to another destination file */ +copyfile(NULL, "/tmp/car", s, COPYFILE_ALL); +copyfile_state_free(s); +/* Remove extended attributes from a file */ +copyfile("/dev/null", "/tmp/bar", NULL, COPYFILE_XATTR); +.Ed .Sh ERRORS .Fn copyfile and @@ -688,40 +726,18 @@ parameter. In addition, both functions may set .Dv errno via an underlying library or system call. -.Sh EXAMPLES -.Bd -literal -offset indent -/* Initialize a state variable */ -copyfile_state_t s; -s = copyfile_state_alloc(); -/* Copy the data and extended attributes of one file to another */ -copyfile("/tmp/f1", "/tmp/f2", s, COPYFILE_DATA | COPYFILE_XATTR); -/* Convert a file to an AppleDouble file for serialization */ -copyfile("/tmp/f2", "/tmp/tmpfile", NULL, COPYFILE_ALL | COPYFILE_PACK); -/* Release the state variable */ -copyfile_state_free(s); -/* A more complex way to call copyfile() */ -s = copyfile_state_alloc(); -copyfile_state_set(s, COPYFILE_STATE_SRC_FILENAME, "/tmp/foo"); -/* One of src or dst must be set... rest can come from the state */ -copyfile(NULL, "/tmp/bar", s, COPYFILE_ALL); -/* Now copy the same source file to another destination file */ -copyfile(NULL, "/tmp/car", s, COPYFILE_ALL); -copyfile_state_free(s); -/* Remove extended attributes from a file */ -copyfile("/dev/null", "/tmp/bar", NULL, COPYFILE_XATTR); -.Ed .Sh SEE ALSO -.Xr listxattr 2 , .Xr getxattr 2 , +.Xr listxattr 2 , .Xr setxattr 2 , .Xr acl 3 +.Sh HISTORY +The +.Fn copyfile +API was introduced in Mac OS X 10.5. .Sh BUGS Both .Fn copyfile functions lack a way to set the input or output block size. .Pp Recursive copies do not honor hard links. -.Sh HISTORY -The -.Fn copyfile -API was introduced in Mac OS X 10.5. diff --git a/copyfile.c b/copyfile.c index 443dbc5..a745d17 100644 --- a/copyfile.c +++ b/copyfile.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2004-2019 Apple, Inc. All rights reserved. + * Copyright (c) 2004-2020 Apple, Inc. All rights reserved. * * @APPLE_LICENSE_HEADER_START@ * @@ -202,6 +202,41 @@ does_copy_protection(int fd) return ((sfs.f_flags & MNT_CPROTECT) == MNT_CPROTECT); } +static bool +path_does_copy_protection(const char *path) +{ + struct statfs sfs; + + if (statfs(path, &sfs) == -1) { + char parent_path[MAXPATHLEN]; + + if (errno != ENOENT) + return false; + + // If the path doesn't exist, + // try to get its parent path and re-attempt the statfs(). + if (dirname_r(path, parent_path) == NULL) + return false; + + if (statfs(parent_path, &sfs) == -1) + return false; + } + + return ((sfs.f_flags & MNT_CPROTECT) == MNT_CPROTECT); +} + +static int +do_copy_protected_open(const char *path, int flags, int class, int dpflags, int mode) +{ + // The passed-in protection class is meaningful, so use open_dprotected_np(). + if (path_does_copy_protection(path)) { + return open_dprotected_np(path, flags, class, dpflags, mode); + } + + // Fall-back to regular open(). + return open(path, flags, mode); +} + static void sort_xattrname_list(void *start, size_t length) { @@ -1077,6 +1112,74 @@ static int copyfile_clone(copyfile_state_t state) return ret; } +/* + * Check if two provided paths are identical, + * and if we're able to determine that, return true. + */ +static bool copyfile_paths_identical(const char *src, const char *dst) +{ + struct attrlist attrs; + struct statfs sfs; + struct stat src_sb, dst_sb; + char volroot[MAXPATHLEN + 1]; + struct { + uint32_t length; + vol_capabilities_attr_t volAttrs; + } volattrs; + char *real_src_path = NULL, *real_dst_path = NULL; + + // Common case: the destination does not exist. + if ((stat(dst, &dst_sb) == -1) || (stat(src, &src_sb) == -1)) + return false; + + // If both files exist, then we next try to check file IDs. + // This requires that the underlying filesystem support persistent file IDs. + if (statfs(src, &sfs) == -1) + return false; + + strlcpy(volroot, sfs.f_mntonname, sizeof(volroot)); + memset(&attrs, 0, sizeof(attrs)); + attrs.bitmapcount = ATTR_BIT_MAP_COUNT; + attrs.volattr = ATTR_VOL_CAPABILITIES; + + if (getattrlist(volroot, &attrs, &volattrs, sizeof(volattrs), 0) == -1) + return false; + + // If the underlying devices are not the same, then the files are not the same. + if (src_sb.st_dev != dst_sb.st_dev) + return false; + + if ((volattrs.volAttrs.capabilities[VOL_CAPABILITIES_FORMAT] & VOL_CAP_FMT_PERSISTENTOBJECTIDS) && + (volattrs.volAttrs.valid[VOL_CAPABILITIES_FORMAT] & VOL_CAP_FMT_PERSISTENTOBJECTIDS)) { + // The underlying source filesystem supports persistent file IDs, + // so if our two files have the same file ID on the same device, + // they are identical. + return (src_sb.st_ino == dst_sb.st_ino); + } + + // Finally, if we don't support persistent file ID's, + // we fall back to path comparisons. + real_src_path = realpath(src, NULL); + if (real_src_path == NULL) + goto exit; + + real_dst_path = realpath(dst, NULL); + if (real_dst_path == NULL) + goto exit; + + if (strncasecmp(src, dst, MAXPATHLEN) == 0) + return true; + +exit: + if (real_src_path) + free(real_src_path); + + if (real_dst_path) + free(real_dst_path); + + return false; +} + /* * the original copyfile() routine; this copies a source file to a destination * file. Note that because we need to set the names in the state variable, this @@ -1134,6 +1237,20 @@ int copyfile(const char *src, const char *dst, copyfile_state_t state, copyfile_ COPYFILE_SET_FNAME(src, s); COPYFILE_SET_FNAME(dst, s); + if (!(s->flags & COPYFILE_CHECK)) { + // We have no work to do if `src` and `dst` point to the same place. + if (copyfile_paths_identical(src, dst)) { + // ...but return an error if requested to do so. + if (s->flags & COPYFILE_EXCL) { + s->err = EEXIST; + goto error_exit; + } + + ret = 0; + goto exit; + } + } + if (s->flags & COPYFILE_RECURSIVE) { ret = copytree(s); goto exit; @@ -1678,7 +1795,7 @@ static int copyfile_unset_acl(copyfile_state_t s) */ static int copyfile_open(copyfile_state_t s) { - int oflags = O_EXCL | O_CREAT | O_WRONLY; + int oflags = O_EXCL | O_CREAT; int islnk = 0, isdir = 0, isreg = 0; int osrc = 0, dsrc = 0; int prot_class = PROTECTION_CLASS_DEFAULT; @@ -1749,6 +1866,26 @@ static int copyfile_open(copyfile_state_t s) if (s->dst && s->dst_fd == -2) { + /* + * Per , only open files for writing if we expect + * to modify the file's content. This avoids undesirable side effects + * of O_WRONLY when we're only modifying metadata/attributes. + * + * The calls needed by COPYFILE_METADATA (e.g. fchown(), fchmod(), + * fchflags(), futimes(), fsetattrlist(), f{set,remove}xattr(), and + * acl_set_fd()) are safe to use with O_RDONLY descriptors. They + * operate on the underlying vnode_t, as path-based variants do. + * These usually only need write permissions in st_mode or ACLs. + */ + const copyfile_flags_t writable_flags = (COPYFILE_DATA | COPYFILE_DATA_SPARSE); + if (COPYFILE_PACK & s->flags) { + oflags |= O_WRONLY; // always writes file content + } else if (COPYFILE_UNPACK & s->flags) { + oflags |= O_RDONLY; // only updates metadata + } else { + oflags |= (writable_flags & s->flags) ? O_WRONLY : O_RDONLY; + } + /* * COPYFILE_UNLINK tells us to try removing the destination * before we create it. We don't care if the file doesn't @@ -1841,7 +1978,8 @@ static int copyfile_open(copyfile_state_t s) return -1; } set_cprot_explicit = 1; - } else while((s->dst_fd = open_dprotected_np(s->dst, oflags | dsrc, prot_class, 0, s->sb.st_mode | S_IWUSR)) < 0) + } else while((s->dst_fd = do_copy_protected_open(s->dst, oflags | dsrc, prot_class, + 0, s->sb.st_mode | S_IWUSR)) < 0) { /* * We set S_IWUSR because fsetxattr does not -- at the time this comment @@ -2053,10 +2191,13 @@ static int copyfile_data_sparse(copyfile_state_t s, size_t input_blk_size, size_ if (!(s->flags & COPYFILE_DATA_SPARSE)) { // Don't attempt this unless the right flags are passed. return ENOTSUP; - } else if (src_size <= 0) { + } else if (src_size < 0) { // The file size of our source is invalid; there's nothing to copy. errno = EINVAL; goto error_exit; + } else if (src_size == 0) { + // This is a zero-length file: no work to do. + goto exit; } // Since a major underlying filesystem requires that holes are block-aligned, @@ -2761,6 +2902,8 @@ static int copyfile_stat(copyfile_state_t s) return 0; } +#define MAX_GETXATTR_RETRIES 3 +#define MAX_XATTR_BUFFER_SIZE (32 * 1024 * 1024) // 32 MiB /* * Similar to copyfile_security() in some ways; this * routine copies the extended attributes from the source, @@ -2774,31 +2917,53 @@ static int copyfile_stat(copyfile_state_t s) static int copyfile_xattr(copyfile_state_t s) { char *name; - char *namebuf, *end; + char *namebuf = NULL; + char *end; ssize_t xa_size; void *xa_dataptr; - ssize_t bufsize = 4096; + ssize_t xa_bufsize = 4096; + ssize_t namebuf_size = 0; ssize_t asize; - ssize_t nsize; + ssize_t list_size; int ret = 0; int look_for_decmpea = 0; + int tries_left = MAX_GETXATTR_RETRIES; /* delete EAs on destination */ - if ((nsize = flistxattr(s->dst_fd, 0, 0, 0)) > 0) +dst_restart: + if ((list_size = flistxattr(s->dst_fd, 0, 0, 0)) > 0) { - if ((namebuf = (char *) malloc(nsize)) == NULL) - return -1; - else - nsize = flistxattr(s->dst_fd, namebuf, nsize, 0); + /* this is always true on the first call: no buffer yet (namebuf_size == 0) */ + if (list_size > namebuf_size) { + if (list_size > MAX_XATTR_BUFFER_SIZE) { + copyfile_warn("destination's xattr list size (%zu) exceeds the threshold (%d); trying to allocate", list_size, MAX_XATTR_BUFFER_SIZE); + } + namebuf_size = list_size; + void *tdptr = namebuf; - if (nsize > 0) { + if ((namebuf = + (void *) realloc((void *) namebuf, namebuf_size)) == NULL) + { + if (tdptr) { + free(tdptr); + } + return -1; + } + } + list_size = flistxattr(s->dst_fd, namebuf, namebuf_size, 0); + + if ((list_size < 0) && (errno == ERANGE) && (tries_left > 0)) { + /* `namebuf` is too small - try again */ + tries_left--; + goto dst_restart; + } else if (list_size > 0) { /* * With this, end points to the last byte of the allocated buffer * This *should* be NUL, from flistxattr, but if it's not, we can * set it anyway -- it'll result in a truncated name, which then * shouldn't match when we get them later. */ - end = namebuf + nsize - 1; + end = namebuf + list_size - 1; if (*end != 0) *end = 0; for (name = namebuf; name <= end; name += strlen(name) + 1) { @@ -2809,15 +2974,24 @@ static int copyfile_xattr(copyfile_state_t s) fremovexattr(s->dst_fd, name,0); } } - free(namebuf); - } else - if (nsize < 0) - { - if (errno == ENOTSUP || errno == EPERM) - return 0; - else - return -1; + } + else if (list_size < 0) + { + if (namebuf) { + free(namebuf); } + if (errno == ENOTSUP || errno == EPERM) { + return 0; + } else { + return -1; + } + } + if (namebuf) { + free(namebuf); + } + namebuf = NULL; + namebuf_size = 0; + tries_left = MAX_GETXATTR_RETRIES; #ifdef DECMPFS_XATTR_NAME if ((s->flags & COPYFILE_DATA) && @@ -2829,24 +3003,50 @@ static int copyfile_xattr(copyfile_state_t s) #endif /* get name list of EAs on source */ - if ((nsize = flistxattr(s->src_fd, 0, 0, look_for_decmpea)) < 0) +src_restart: + if ((list_size = flistxattr(s->src_fd, 0, 0, look_for_decmpea)) <= 0) { - if (errno == ENOTSUP || errno == EPERM) + if (namebuf) { + free(namebuf); + } + + if (list_size == 0) { return 0; - else + } else if (errno == ENOTSUP || errno == EPERM) { + return 0; + } else { return -1; - } else - if (nsize == 0) - return 0; + } + } - if ((namebuf = (char *) malloc(nsize)) == NULL) - return -1; - else - nsize = flistxattr(s->src_fd, namebuf, nsize, look_for_decmpea); + /* this is always true on the first call: no buffer yet (namebuf_size == 0) */ + if (list_size > namebuf_size) { + if (list_size > MAX_XATTR_BUFFER_SIZE) { + copyfile_warn("source's xattr list size (%zu) exceeds the threshold (%d); trying to allocate", list_size, MAX_XATTR_BUFFER_SIZE); + } + namebuf_size = list_size; + void *tdptr = namebuf; + if ((namebuf = + (void *) realloc((void *) namebuf, namebuf_size)) == NULL) + { + if (tdptr) { + free(tdptr); + } + return -1; + } + } - if (nsize <= 0) { - free(namebuf); - return (int)nsize; + list_size = flistxattr(s->src_fd, namebuf, namebuf_size, look_for_decmpea); + + if ((list_size < 0) && (errno == ERANGE) && (tries_left > 0)) { + /* `namebuf` is too small - try again */ + tries_left--; + goto src_restart; + } else if (list_size <= 0) { + if (namebuf) { + free(namebuf); + } + return (int)list_size; } /* @@ -2855,11 +3055,11 @@ static int copyfile_xattr(copyfile_state_t s) * set it anyway -- it'll result in a truncated name, which then * shouldn't match when we get them later. */ - end = namebuf + nsize - 1; + end = namebuf + list_size - 1; if (*end != 0) *end = 0; - if ((xa_dataptr = (void *) malloc(bufsize)) == NULL) { + if ((xa_dataptr = (void *) malloc(xa_bufsize)) == NULL) { free(namebuf); return -1; } @@ -2875,17 +3075,22 @@ static int copyfile_xattr(copyfile_state_t s) if (strncmp(name, XATTR_QUARANTINE_NAME, end - name) == 0) continue; + tries_left = MAX_GETXATTR_RETRIES; +get_restart: if ((xa_size = fgetxattr(s->src_fd, name, 0, 0, 0, look_for_decmpea)) < 0) { continue; } - if (xa_size > bufsize) + if (xa_size > xa_bufsize) { + if (xa_size > MAX_XATTR_BUFFER_SIZE) { + copyfile_warn("xattr named %s has size (%zu), which exceeds the threshold (%d); trying to allocate", name, list_size, MAX_XATTR_BUFFER_SIZE); + } void *tdptr = xa_dataptr; - bufsize = xa_size; + xa_bufsize = xa_size; if ((xa_dataptr = - (void *) realloc((void *) xa_dataptr, bufsize)) == NULL) + (void *) realloc((void *) xa_dataptr, xa_bufsize)) == NULL) { free(tdptr); ret = -1; @@ -2893,8 +3098,13 @@ static int copyfile_xattr(copyfile_state_t s) } } - if ((asize = fgetxattr(s->src_fd, name, xa_dataptr, xa_size, 0, look_for_decmpea)) < 0) + if ((asize = fgetxattr(s->src_fd, name, xa_dataptr, xa_bufsize, 0, look_for_decmpea)) < 0) { + if ((errno == ERANGE) && (tries_left > 0)) { + /* `xa_dataptr` is too small - try again */ + tries_left--; + goto get_restart; + } continue; } diff --git a/copyfile.xcodeproj/project.pbxproj b/copyfile.xcodeproj/project.pbxproj index f282750..4cd88ec 100644 --- a/copyfile.xcodeproj/project.pbxproj +++ b/copyfile.xcodeproj/project.pbxproj @@ -7,7 +7,9 @@ objects = { /* Begin PBXBuildFile section */ + 096213F7239827D0005847FC /* identical_test.c in Sources */ = {isa = PBXBuildFile; fileRef = 096213F6239827D0005847FC /* identical_test.c */; }; 098AF3B622692BF300F9BA42 /* stat_test.c in Sources */ = {isa = PBXBuildFile; fileRef = 098AF3B522692BF300F9BA42 /* stat_test.c */; }; + 3EF9FA5F2418B6BA003B43E8 /* readonly_fd_test.c in Sources */ = {isa = PBXBuildFile; fileRef = 3EF9FA5E2418B6BA003B43E8 /* readonly_fd_test.c */; }; 721D4F071EA95283000F0555 /* copyfile.c in Sources */ = {isa = PBXBuildFile; fileRef = FCCE17C1135A658F002CEE6D /* copyfile.c */; }; 721D4F081EA95290000F0555 /* xattr_flags.c in Sources */ = {isa = PBXBuildFile; fileRef = 72406E621676C3C80099568B /* xattr_flags.c */; }; 72406E631676C3C80099568B /* xattr_flags.c in Sources */ = {isa = PBXBuildFile; fileRef = 72406E621676C3C80099568B /* xattr_flags.c */; }; @@ -18,6 +20,7 @@ 72B4C0F41676C47D00C13E05 /* copyfile_private.h in Headers */ = {isa = PBXBuildFile; fileRef = 72B4C0F31676C47D00C13E05 /* copyfile_private.h */; settings = {ATTRIBUTES = (Private, ); }; }; 72EAA3B016A72F4500833E98 /* xattr_flags.h in Headers */ = {isa = PBXBuildFile; fileRef = 72EAA3AF16A72F4500833E98 /* xattr_flags.h */; settings = {ATTRIBUTES = (Public, ); }; }; 86EF9F0A1834018C00AAB3F3 /* xattr_properties.h in Headers */ = {isa = PBXBuildFile; fileRef = 86EF9F091834018C00AAB3F3 /* xattr_properties.h */; settings = {ATTRIBUTES = (Private, ); }; }; + D11048A22455AE7900E8F465 /* xattr_test.c in Sources */ = {isa = PBXBuildFile; fileRef = D11048A12455AE7900E8F465 /* xattr_test.c */; }; FCCE17C3135A658F002CEE6D /* copyfile.c in Sources */ = {isa = PBXBuildFile; fileRef = FCCE17C1135A658F002CEE6D /* copyfile.c */; }; FCCE17C4135A658F002CEE6D /* copyfile.h in Headers */ = {isa = PBXBuildFile; fileRef = FCCE17C2135A658F002CEE6D /* copyfile.h */; settings = {ATTRIBUTES = (Public, ); }; }; /* End PBXBuildFile section */ @@ -35,9 +38,13 @@ /* End PBXCopyFilesBuildPhase section */ /* Begin PBXFileReference section */ + 096213F5239827D0005847FC /* identical_test.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = identical_test.h; sourceTree = ""; }; + 096213F6239827D0005847FC /* identical_test.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = identical_test.c; sourceTree = ""; }; 098AF3B422692BF300F9BA42 /* stat_test.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = stat_test.h; sourceTree = ""; }; 098AF3B522692BF300F9BA42 /* stat_test.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = stat_test.c; sourceTree = ""; }; 098AF3B7226A510E00F9BA42 /* copyfile_test.entitlements */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.entitlements; path = copyfile_test.entitlements; sourceTree = ""; }; + 3EF9FA5D2418B6BA003B43E8 /* readonly_fd_test.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = readonly_fd_test.h; sourceTree = ""; }; + 3EF9FA5E2418B6BA003B43E8 /* readonly_fd_test.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = readonly_fd_test.c; sourceTree = ""; }; 3F1EFD4C185C4EB400D1C970 /* copyfile.xcconfig */ = {isa = PBXFileReference; lastKnownFileType = text.xcconfig; name = copyfile.xcconfig; path = xcodescripts/copyfile.xcconfig; sourceTree = ""; }; 721D4F051EA95008000F0555 /* libcopyfile.tbd */ = {isa = PBXFileReference; lastKnownFileType = "sourcecode.text-based-dylib-definition"; name = libcopyfile.tbd; path = Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/lib/system/libcopyfile.tbd; sourceTree = DEVELOPER_DIR; }; 72406E621676C3C80099568B /* xattr_flags.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = xattr_flags.c; sourceTree = ""; }; @@ -53,6 +60,8 @@ 72EAA3AF16A72F4500833E98 /* xattr_flags.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = xattr_flags.h; sourceTree = ""; }; 861E1C14180F0AF900E65B9A /* xattr_name_with_flags.3 */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = xattr_name_with_flags.3; sourceTree = ""; }; 86EF9F091834018C00AAB3F3 /* xattr_properties.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = xattr_properties.h; sourceTree = ""; }; + D11048A02455AE7900E8F465 /* xattr_test.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = xattr_test.h; sourceTree = ""; }; + D11048A12455AE7900E8F465 /* xattr_test.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = xattr_test.c; sourceTree = ""; }; FCCE17BB135A6444002CEE6D /* libcopyfile.dylib */ = {isa = PBXFileReference; explicitFileType = "compiled.mach-o.dylib"; includeInIndex = 0; path = libcopyfile.dylib; sourceTree = BUILT_PRODUCTS_DIR; }; FCCE17C0135A658F002CEE6D /* copyfile.3 */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = copyfile.3; sourceTree = ""; }; FCCE17C1135A658F002CEE6D /* copyfile.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = copyfile.c; sourceTree = ""; }; @@ -89,6 +98,10 @@ isa = PBXGroup; children = ( 726EE9DA1E9423E50017A5B9 /* main.c */, + 096213F6239827D0005847FC /* identical_test.c */, + 096213F5239827D0005847FC /* identical_test.h */, + 3EF9FA5E2418B6BA003B43E8 /* readonly_fd_test.c */, + 3EF9FA5D2418B6BA003B43E8 /* readonly_fd_test.h */, 726EE9DE1E9425160017A5B9 /* sparse_test.c */, 726EE9DF1E9425160017A5B9 /* sparse_test.h */, 098AF3B522692BF300F9BA42 /* stat_test.c */, @@ -98,6 +111,8 @@ 726EE9E21E946B320017A5B9 /* systemx.c */, 726EE9E31E946B320017A5B9 /* systemx.h */, 098AF3B7226A510E00F9BA42 /* copyfile_test.entitlements */, + D11048A02455AE7900E8F465 /* xattr_test.h */, + D11048A12455AE7900E8F465 /* xattr_test.c */, ); path = copyfile_test; sourceTree = ""; @@ -201,6 +216,7 @@ developmentRegion = English; hasScannedForEncodings = 0; knownRegions = ( + English, en, ); mainGroup = FCCE17AB135A5FFB002CEE6D; @@ -238,9 +254,12 @@ files = ( 098AF3B622692BF300F9BA42 /* stat_test.c in Sources */, 721D4F081EA95290000F0555 /* xattr_flags.c in Sources */, + 3EF9FA5F2418B6BA003B43E8 /* readonly_fd_test.c in Sources */, + D11048A22455AE7900E8F465 /* xattr_test.c in Sources */, 721D4F071EA95283000F0555 /* copyfile.c in Sources */, 726EE9DB1E9423E50017A5B9 /* main.c in Sources */, 726EE9E61E946D590017A5B9 /* test_utils.c in Sources */, + 096213F7239827D0005847FC /* identical_test.c in Sources */, 726EE9E41E946B320017A5B9 /* systemx.c in Sources */, 726EE9E01E9425160017A5B9 /* sparse_test.c in Sources */, ); @@ -366,6 +385,7 @@ "-ldispatch", "-lxpc", ); + OTHER_TAPI_FLAGS = "-umbrella System"; SDKROOT = macosx.internal; "SIM_SUFFIX[sdk=iphonesimulator*]" = _sim; SUPPORTS_TEXT_BASED_API = YES; diff --git a/copyfile_test/identical_test.c b/copyfile_test/identical_test.c new file mode 100644 index 0000000..19f8dd6 --- /dev/null +++ b/copyfile_test/identical_test.c @@ -0,0 +1,119 @@ +// +// identical_test.c +// copyfile_test +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../copyfile.h" +#include "identical_test.h" +#include "test_utils.h" + +#define REGULAR_FILE_NAME "regular_file" +#define REGULAR_DIR_NAME "regular_dir" +#define DUMMY_XATTR_NAME "dummy_xattr" +#define DUMMY_XATTR_DATA "drell" +#define TEST_FILE_DATA "krogan" + +static bool verify_src_dst_identical(const char *apfs_test_directory, __unused size_t block_size) { + char regular_file[BSIZE_B] = {0}, folder[BSIZE_B] = {0}, file_inside_folder[BSIZE_B] = {0}; + int regular_file_fd, file_inside_folder_fd; + bool success = true; + + // The idea here is to verify that copyfile(file1, file1) returns success + // without doing anything. + // There are a few wrinkles - COPYFILE_CHECK still needs to work on these files, + // and we need to make sure that our identity check works on filesystems without + // persistent object identifiers. The first we can easily verify but the second + // is not tested here. Nor are negative tests included (there are an infinite + // number of those, so we rely on the other tests to verify that behavior). + + // Create path names. + assert_with_errno(snprintf(regular_file, BSIZE_B, "%s/" REGULAR_FILE_NAME, apfs_test_directory) > 0); + assert_with_errno(snprintf(folder, BSIZE_B, "%s/" REGULAR_DIR_NAME, apfs_test_directory) > 0); + assert_with_errno(snprintf(file_inside_folder, BSIZE_B, "%s/" REGULAR_FILE_NAME, folder) > 0); + + // First, verify copyfile(file1, file1), + // where file1 is a regular file. + + // Create our regular file. + regular_file_fd = open(regular_file, DEFAULT_OPEN_FLAGS, DEFAULT_OPEN_PERM); + assert_with_errno(regular_file_fd >= 0); + + // Write some data to the test file so that we can verify it is not empty. + assert(write(regular_file_fd, TEST_FILE_DATA, sizeof(TEST_FILE_DATA)) > 0); + + // Verify copyfile(file1, file1) does nothing. + assert_no_err(copyfile(regular_file, regular_file, NULL, COPYFILE_ALL)); + success = success && verify_contents_with_buf(regular_file_fd, 0, (const char *)TEST_FILE_DATA, sizeof(TEST_FILE_DATA)); + + // Verify copyfile(file1, file1, COPYFILE_EXCL) returns an error. + assert(copyfile(regular_file, regular_file, NULL, COPYFILE_ALL|COPYFILE_EXCL) == -1); + assert(errno == EEXIST); + + // Write an dummy xattr to the file to verify COPYFILE_CHECK. + assert_no_err(fsetxattr(regular_file_fd, DUMMY_XATTR_NAME, DUMMY_XATTR_DATA, sizeof(DUMMY_XATTR_DATA), 0, XATTR_CREATE)); + + // Verify copyfile(file1, file1, ..., COPYFILE_CHECK) works. + assert_no_err(copyfile(regular_file, regular_file, NULL, COPYFILE_CHECK) == COPYFILE_XATTR); + + // Now, verify that copyfile(dir1, dir1, COPYFILE_RECURSIVE) + // also returns early. Do this by making sure the contents of a file inside the directory + // do not change after copyfile(COPYFILE_RECURSIVE). + + // Create our directory. + assert_no_err(mkdir(folder, DEFAULT_MKDIR_PERM)); + + // Create a regular file inside that directory. + + file_inside_folder_fd = open(file_inside_folder, DEFAULT_OPEN_FLAGS, DEFAULT_OPEN_PERM); + assert_with_errno(file_inside_folder_fd >= 0); + + // Write some data to the interior file so that we can verify it is not empty. + assert(write(file_inside_folder_fd, (const char *)TEST_FILE_DATA, sizeof(TEST_FILE_DATA)) > 0); + + // Verify copyfile(dir1, dir1, ... COPYFILE_RECURSIVE). + assert_no_err(copyfile(folder, folder, NULL, COPYFILE_RECURSIVE)); + success = success && verify_contents_with_buf(file_inside_folder_fd, 0, TEST_FILE_DATA, sizeof(TEST_FILE_DATA)); + + // Post-test cleanup. + assert_no_err(close(file_inside_folder_fd)); + assert_no_err(close(regular_file_fd)); + (void)removefile(folder, NULL, REMOVEFILE_RECURSIVE); + (void)removefile(regular_file, NULL, 0); + + return success; +} + +bool do_src_dst_identical_test(const char *apfs_test_directory, __unused size_t block_size) { + char test_dir[BSIZE_B] = {0}; + int test_folder_id; + bool success = true; + + printf("START [identical]\n"); + + // Get ready for the test. + test_folder_id = rand() % DEFAULT_NAME_MOD; + create_test_file_name(apfs_test_directory, "identical", test_folder_id, test_dir); + assert_no_err(mkdir(test_dir, DEFAULT_MKDIR_PERM)); + + success = verify_src_dst_identical(test_dir, block_size); + + if (success) { + printf("PASS [identical]\n"); + } else { + printf("FAIL [identical]\n"); + } + + (void)removefile(test_dir, NULL, REMOVEFILE_RECURSIVE); + + return success ? EXIT_SUCCESS : EXIT_FAILURE; +} \ No newline at end of file diff --git a/copyfile_test/identical_test.h b/copyfile_test/identical_test.h new file mode 100644 index 0000000..e4085bd --- /dev/null +++ b/copyfile_test/identical_test.h @@ -0,0 +1,14 @@ +// +// identical_test.h +// copyfile_test +// + +#ifndef identical_test_h +#define identical_test_h + +#include +#include + +bool do_src_dst_identical_test(const char *apfs_test_directory, size_t block_size); + +#endif /* identical_test_h */ diff --git a/copyfile_test/main.c b/copyfile_test/main.c index a5384e9..af5ec06 100644 --- a/copyfile_test/main.c +++ b/copyfile_test/main.c @@ -11,8 +11,11 @@ #include #include +#include "identical_test.h" +#include "readonly_fd_test.h" #include "sparse_test.h" #include "stat_test.h" +#include "xattr_test.h" #include "test_utils.h" #define DISK_IMAGE_SIZE_MB 512 @@ -51,11 +54,14 @@ int main(__unused int argc, __unused const char * argv[]) { // Run our tests. sranddev(); + failed |= do_readonly_fd_test(TEST_DIR, stb.f_bsize); failed |= do_sparse_test(TEST_DIR, stb.f_bsize); failed |= do_sparse_recursive_test(TEST_DIR, stb.f_bsize); failed |= do_fcopyfile_offset_test(TEST_DIR, stb.f_bsize); failed |= do_preserve_dst_flags_test(TEST_DIR, stb.f_bsize); failed |= do_preserve_dst_tracked_test(TEST_DIR, stb.f_bsize); + failed |= do_src_dst_identical_test(TEST_DIR, stb.f_bsize); + failed |= do_xattr_test(TEST_DIR, stb.f_bsize); // Cleanup the disk image we ran our tests on. if (USING_DISK_IMAGE) { diff --git a/copyfile_test/readonly_fd_test.c b/copyfile_test/readonly_fd_test.c new file mode 100644 index 0000000..ee5a0b3 --- /dev/null +++ b/copyfile_test/readonly_fd_test.c @@ -0,0 +1,124 @@ +// +// Copyright (c) 2020 Apple Inc. All rights reserved. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "readonly_fd_test.h" +#include "test_utils.h" + + +static +bool test_readonly_fd_metadata(const char *basedir) +{ + char filename[] = ".readonly-ops-XXXXXX"; + bool created = false; + bool success = true; + int dirfd = -1; + int tmpfd = -1; + int fd = -1; + acl_t acl = NULL; + + static const char test_name[] = "readonly_fd_metadata"; + printf("START [%s]\n", test_name); + + assert_with_errno((dirfd = open(basedir, O_RDONLY | O_DIRECTORY)) != -1); + assert_with_errno((tmpfd = mkstempsat_np(dirfd, filename, 0)) != -1); + created = true; + + assert_with_errno((fd = openat(dirfd, filename, O_RDONLY)) != -1); + close(tmpfd); + tmpfd = -1; + + // confirm that writes are disallowed + const char data[] = "failure"; + assert(write(fd, data, sizeof(data) - 1) == -1); + + // check fchown() + const uid_t uid = geteuid(); + const gid_t gid = getegid(); + assert_no_err(fchown(fd, uid, gid)); + + // check fchmod() + assert_no_err(fchmod(fd, 0644)); + assert_no_err(fchmod(fd, 0600)); + + // check fchflags() + assert_no_err(fchflags(fd, UF_HIDDEN)); + + // check setting timestamps with fsetattrlist + const time_t mtime = 978307200; + const time_t atime = mtime + 1; + + struct timeval matimes_usec[] = {{mtime, 0}, {atime, 0}}; + assert_no_err(futimes(fd, matimes_usec)); + + struct attrlist attrlist = { + .bitmapcount = ATTR_BIT_MAP_COUNT, + .commonattr = ATTR_CMN_MODTIME | ATTR_CMN_ACCTIME, + }; + struct { + struct timespec mtime; + struct timespec atime; + } matimes_nsec = {{mtime, 0}, {atime, 0}}; + assert_no_err(fsetattrlist(fd, &attrlist, &matimes_nsec, sizeof(matimes_nsec), 0)); + + // check adding and removing xattrs + static const char key[] = "local.test-xattr"; + static const char value[] = "local.test-xattr.value"; + assert_no_err(fsetxattr(fd, key, value, sizeof(value)-1, 0, 0)); + assert_no_err(fremovexattr(fd, key, 0)); + + // check setting ACLs + assert_with_errno((acl = acl_init(1)) != NULL); + assert_no_err(acl_set_fd(fd, acl)); + + // log pass/fail before cleanup + if (success) { + printf("PASS [%s]\n", test_name); + } else { + printf("FAIL [%s]\n", test_name); + } + + // clean up resources + if (acl) { + acl_free(acl); + } + if (fd != -1) { + close(fd); + } + if (tmpfd != -1) { + close(tmpfd); + } + if (created) { + unlinkat(dirfd, filename, 0); + } + if (dirfd != -1) { + close(dirfd); + } + + return success; +} + + +bool do_readonly_fd_test(const char *apfs_test_directory, size_t block_size __unused) +{ + // These tests verify the underlying calls needed for COPYFILE_METADATA + // operations are safe with O_RDONLY file descriptors. If this fails, + // expect to cause many other copyfile() failures. + bool success = true; + success = success && test_readonly_fd_metadata(apfs_test_directory); + return !success; // caller expects nonzero to mean failure +} diff --git a/copyfile_test/readonly_fd_test.h b/copyfile_test/readonly_fd_test.h new file mode 100644 index 0000000..339230f --- /dev/null +++ b/copyfile_test/readonly_fd_test.h @@ -0,0 +1,13 @@ +// +// Copyright (c) 2020 Apple Inc. All rights reserved. +// + +#ifndef readonly_fd_test_h +#define readonly_fd_test_h + +#include +#include + +bool do_readonly_fd_test(const char *apfs_test_directory, size_t block_size); + +#endif /* readonly_fd_test_h */ diff --git a/copyfile_test/sparse_test.c b/copyfile_test/sparse_test.c index fc126d7..0a75df5 100644 --- a/copyfile_test/sparse_test.c +++ b/copyfile_test/sparse_test.c @@ -182,12 +182,16 @@ static off_t write_diff_adj_holes(int fd, off_t block_size) { return 0; } +static off_t write_nothing(__unused int fd, __unused off_t block_size) { + return 0; +} + typedef struct { creator_func func; // pointer to function to create a sparse file const char * name; // null terminated string } sparse_test_func; -#define NUM_TEST_FUNCTIONS 10 +#define NUM_TEST_FUNCTIONS 11 sparse_test_func test_functions[NUM_TEST_FUNCTIONS] = { {write_start_and_end_holes, "start_and_end_holes"}, {write_middle_hole, "middle_hole"}, @@ -198,7 +202,8 @@ sparse_test_func test_functions[NUM_TEST_FUNCTIONS] = { {write_no_sparse, "no_sparse"}, {write_sparse_odd_offset, "write_sparse_odd_offset"}, {write_sparse_bs_offset, "write_sparse_bs_offset"}, - {write_diff_adj_holes, "write_diff_adj_holes"} + {write_diff_adj_holes, "write_diff_adj_holes"}, + {write_nothing, "write_nothing"}, }; bool do_sparse_test(const char* apfs_test_directory, size_t block_size) { diff --git a/copyfile_test/stat_test.c b/copyfile_test/stat_test.c index 0ab2944..784b046 100644 --- a/copyfile_test/stat_test.c +++ b/copyfile_test/stat_test.c @@ -174,8 +174,8 @@ bool do_preserve_dst_tracked_test(const char *test_directory, __unused size_t bl assert_no_err(copyfile(file_src, file_dst, NULL, COPYFILE_DATA|COPYFILE_STAT|COPYFILE_PRESERVE_DST_TRACKED)); assert_no_err(stat(file_dst, &dst_stb)); - success &= (dst_stb.st_size == src_fsize); - success &= (dst_stb.st_flags & UF_TRACKED); + success = success && (dst_stb.st_size == src_fsize); + success = success && (dst_stb.st_flags & UF_TRACKED); if (success) { printf("PASS [preserve_dst_tracked]\n"); } else { diff --git a/copyfile_test/test_utils.c b/copyfile_test/test_utils.c index 95741b1..0da0b87 100644 --- a/copyfile_test/test_utils.c +++ b/copyfile_test/test_utils.c @@ -11,10 +11,78 @@ #include #include #include +#include #include "test_utils.h" #include "systemx.h" +static bool verify_xattr_content(int fd, const char *xattr_name, const char *expected, ssize_t size) { + // Verify that the file referenced by `fd` has an xattr named `xattr_name` + // of size `size` and with contents equal to `expected`. + char *actual = NULL; + bool equal; + + assert(fd > 0 && xattr_name && expected); + assert_with_errno(actual = malloc(size)); + assert_with_errno(fgetxattr(fd, xattr_name, actual, size, 0, 0) == size); + + equal = (memcmp(actual, expected, size) == 0); + if (!equal) { + printf("xattr %s: content does not match expected\n", xattr_name); + } + + free(actual); + return equal; +} + +bool verify_fd_xattr_contents(int orig_fd, int copy_fd) { + // Verify that both fd's have the same xattrs. + // We do so by first verifying that `flistxattr()` returns the same size + // for both, and then validating that `copy_fd` has each of the xattrs + // that `orig_fd` has. + char *namebuf = NULL, *xa_buf = NULL, *name, *end; + ssize_t orig_size, copy_size, xa_size; + bool equal = true; + + assert((orig_fd > 0) && (copy_fd > 0)); + + orig_size = flistxattr(orig_fd, 0, 0, XATTR_SHOWCOMPRESSION); + copy_size = flistxattr(copy_fd, 0, 0, XATTR_SHOWCOMPRESSION); + if (orig_size != copy_size) { + printf("xattrlist size: orig_size(%zu) != (%zu)copy_size\n", orig_size, copy_size); + return false; + } + + if (orig_size == 0) { + return true; + } + + assert_with_errno(namebuf = malloc(orig_size)); + + assert_with_errno(flistxattr(orig_fd, namebuf, orig_size, 0) == orig_size); + + end = namebuf + orig_size - 1; + if (*end != 0) { + *end = 0; + } + + for (name = namebuf; name <= end; name += strlen(name) + 1) { + xa_size = fgetxattr(orig_fd, name, 0, 0, 0, 0); + assert(xa_size >= 0); + assert_with_errno(xa_buf = malloc(xa_size)); + assert_with_errno(fgetxattr(orig_fd, name, xa_buf, xa_size, 0, 0) == xa_size); + equal = equal && verify_xattr_content(copy_fd, name, xa_buf, xa_size); + free(xa_buf); + + if (!equal) { + break; + } + } + free(namebuf); + + return equal; +} + bool verify_st_flags(struct stat *sb, uint32_t flags_to_expect) { // Verify that sb's flags include flags_to_expect. if (((sb->st_flags & flags_to_expect)) != flags_to_expect) { @@ -26,6 +94,37 @@ bool verify_st_flags(struct stat *sb, uint32_t flags_to_expect) { return true; } +bool verify_contents_with_buf(int orig_fd, off_t orig_pos, const char *expected, size_t length) +{ + // Read *length* bytes from a file descriptor at a specified position + // and assert that they match *length* bytes from expected. + char orig_contents[length]; + bool equal; + + assert(orig_fd > 0 && orig_pos >= 0); + memset(orig_contents, 0, length); + + errno = 0; + ssize_t pread_res = pread(orig_fd, orig_contents, length, orig_pos); + assert_with_errno(pread_res == (off_t) length); + equal = (memcmp(orig_contents, expected, length) == 0); + if (!equal) { + printf("fd (%lld - %lld) did not match expected contents\n", orig_pos, orig_pos + length); + + // Find the first non-matching byte and print it out. + for (size_t bad_off = 0; bad_off < length; bad_off++) { + if (orig_contents[bad_off] != expected[bad_off]) { + printf("first mismatch is at offset %zu, original 0x%llx expected 0x%llx\n", + bad_off, (unsigned long long)orig_contents[bad_off], + (unsigned long long)expected[bad_off]); + break; + } + } + } + + return equal; +} + bool verify_fd_contents(int orig_fd, off_t orig_pos, int copy_fd, off_t copy_pos, size_t length) { // Read *length* contents of the two fds and make sure they compare as equal. // Don't alter the position of either fd. @@ -33,6 +132,7 @@ bool verify_fd_contents(int orig_fd, off_t orig_pos, int copy_fd, off_t copy_pos bool equal; assert(orig_fd > 0 && copy_fd > 0); + assert(orig_pos >= 0); memset(orig_contents, 0, length); memset(copy_contents, 0, length); @@ -119,7 +219,7 @@ int create_hole_in_fd(int fd, off_t offset, off_t length) { void create_test_file_name(const char *dir, const char *postfix, int id, char *string_out) { - // Make a name for this new file and put it in out_name, which should be BSIZE_B bytes. + // Make a name for this new file and put it in string_out, which should be BSIZE_B bytes. assert_with_errno(snprintf(string_out, BSIZE_B, "%s/testfile-%d.%s", dir, id, postfix) > 0); } diff --git a/copyfile_test/test_utils.h b/copyfile_test/test_utils.h index 4cd7faa..decbe1e 100644 --- a/copyfile_test/test_utils.h +++ b/copyfile_test/test_utils.h @@ -35,7 +35,9 @@ #define DIFF_PATH "/usr/bin/diff" // Test routine helpers. +bool verify_fd_xattr_contents(int orig_fd, int copy_fd); bool verify_st_flags(struct stat *sb, uint32_t flags_to_expect); +bool verify_contents_with_buf(int orig_fd, off_t orig_pos, const char *expected, size_t length); bool verify_fd_contents(int orig_fd, off_t orig_pos, int copy_fd, off_t copy_pos, size_t length); bool verify_copy_contents(const char *orig_name, const char *copy_name); bool verify_copy_sizes(struct stat *orig_sb, struct stat *copy_sb, copyfile_state_t cpf_state, diff --git a/copyfile_test/xattr_test.c b/copyfile_test/xattr_test.c new file mode 100644 index 0000000..472a4a7 --- /dev/null +++ b/copyfile_test/xattr_test.c @@ -0,0 +1,84 @@ +// +// xattr_test.c +// copyfile_test +// + +#include +#include +#include +#include +#include + +#include "xattr_test.h" +#include "test_utils.h" + +#define SRC_FILE_NAME "src_file" +#define DST_FILE_NAME "dst_file" +#define SMALL_XATTR_NAME "small_xattr" +#define SMALL_XATTR_DATA "drell" +#define BIG_XATTR_NAME "big_xattr" +#define BIG_XATTR_SIZE (20 * 1024 * 1024) // 20MiB + +#define DEFAULT_CHAR_MOD 256 + +static bool copy_and_verify_xattr_contents(const char *src_file, const char *dst_file, int src_file_fd, int dst_file_fd) { + assert_no_err(copyfile(src_file, dst_file, NULL, COPYFILE_XATTR)); + + return verify_fd_xattr_contents(src_file_fd, dst_file_fd); +} + +bool do_xattr_test(const char *apfs_test_directory, __unused size_t block_size) { + char test_dir[BSIZE_B] = {0}; + char src_file[BSIZE_B] = {0}, dst_file[BSIZE_B] = {0}; + char *big_xattr_data = NULL, buf[4096] = {0}; + int test_folder_id; + int src_file_fd, dst_file_fd; + bool success = true; + + printf("START [xattr]\n"); + + // Get ready for the test. + test_folder_id = rand() % DEFAULT_NAME_MOD; + create_test_file_name(apfs_test_directory, "xattr", test_folder_id, test_dir); + assert_no_err(mkdir(test_dir, DEFAULT_MKDIR_PERM)); + + // Create path names. + assert_with_errno(snprintf(src_file, BSIZE_B, "%s/" SRC_FILE_NAME, test_dir) > 0); + assert_with_errno(snprintf(dst_file, BSIZE_B, "%s/" DST_FILE_NAME, test_dir) > 0); + + // Create our files. + src_file_fd = open(src_file, DEFAULT_OPEN_FLAGS, DEFAULT_OPEN_PERM); + assert_with_errno(src_file_fd >= 0); + dst_file_fd = open(dst_file, DEFAULT_OPEN_FLAGS, DEFAULT_OPEN_PERM); + assert_with_errno(dst_file_fd >= 0); + + // Sanity check - empty copy + success = success && copy_and_verify_xattr_contents(src_file, dst_file, src_file_fd, dst_file_fd); + + // Write a small xattr to the source file. + assert_no_err(fsetxattr(src_file_fd, SMALL_XATTR_NAME, SMALL_XATTR_DATA, sizeof(SMALL_XATTR_DATA), 0, XATTR_CREATE)); + success = success && copy_and_verify_xattr_contents(src_file, dst_file, src_file_fd, dst_file_fd); + + // Create big xattr data + assert_with_errno(big_xattr_data = malloc(BIG_XATTR_SIZE)); + for (int i = 0; i * sizeof(buf) < BIG_XATTR_SIZE; i++) { + memset(buf, rand() % DEFAULT_CHAR_MOD, sizeof(buf)); + memcpy(big_xattr_data + (i * sizeof(buf)), buf, sizeof(buf)); + } + + // Write a big xattr to the source file. + assert_no_err(fsetxattr(src_file_fd, BIG_XATTR_NAME, big_xattr_data, BIG_XATTR_SIZE, 0, XATTR_CREATE)); + success = success && copy_and_verify_xattr_contents(src_file, dst_file, src_file_fd, dst_file_fd); + + if (success) { + printf("PASS [xattr]\n"); + } else { + printf("FAIL [xattr]\n"); + } + + free(big_xattr_data); + (void)removefile(test_dir, NULL, REMOVEFILE_RECURSIVE); + + return success ? EXIT_SUCCESS : EXIT_FAILURE; +} + diff --git a/copyfile_test/xattr_test.h b/copyfile_test/xattr_test.h new file mode 100644 index 0000000..2749e97 --- /dev/null +++ b/copyfile_test/xattr_test.h @@ -0,0 +1,14 @@ +// +// xattr_test.h +// copyfile_test +// + +#ifndef xattr_test_h +#define xattr_test_h + +#include + +bool do_xattr_test(const char *apfs_test_directory, size_t block_size); + +#endif /* xattr_test_h */ + diff --git a/xattr_name_with_flags.3 b/xattr_name_with_flags.3 index 35afc2a..cd8a3b5 100644 --- a/xattr_name_with_flags.3 +++ b/xattr_name_with_flags.3 @@ -7,6 +7,7 @@ .Sh NAME .Nm xattr_preserve_for_intent , xattr_name_with_flags , xattr_name_without_flags , .Nm xattr_flags_from_name , xattr_intent_with_flags +.Nd obtain properties related to extended attributes, for use in copying .Sh LIBRARY .Lb libc .Sh SYNOPSIS @@ -23,7 +24,8 @@ .Fn xattr_intent_with_flags "xattr_operation_intent_t" "xattr_flags_t" .Sh DESCRIPTION These functions are used in conjunction with copying extended attributes from -one file to another. Various types of copying (an "intent") check flags to +one file to another. +Various types of copying (an "intent") check flags to determine which is allowed or not. .Pp The @@ -32,7 +34,8 @@ function returns an extended attribute name with the appropriate flags encoded as a string; the .Fn xattr_name_without_flags undoes this, giving the name of the extended attribute without the flags -encoding. The slight inverse of that is +encoding. +The slight inverse of that is .Fn xattr_flags_from_name , which will return the flags encoded in a name. .Pp @@ -62,30 +65,30 @@ named extended attribute should be preserved during a copy for the given intent. .Sh INTENT The type -.Dt xattr_operation_intent_t -is an integral type, which is used to indicate what the intent for the operation -is. The following intent values are defined: +.Vt xattr_operation_intent_t +is an integral type, which is used to indicate what the intent for the operation is. +The following intent values are defined: .Bl -tag -width XATTR_OPERATION_INTENT_SHARE .It Dv XATTR_OPERATION_INTENT_COPY Indicates that the intent is to simply copy from the source to the destination. -E.g., with cp. Most extended attributes should generally be preserved in this -case. +E.g., with cp. +Most extended attributes should generally be preserved in this case. .It Dv XATTR_OPERATION_INTENT_SAVE Indicates that intent is to perform a save (perhaps as in a "safe save"). This differs from a copy in that the content may be changing; the destination may be over-writing or replacing the source, and some extended attributes should not be preserved during this process. .It Dv XATTR_OPERATION_INTENT_SHARE -Indicates that the intent is to share, or export, the object. For example, -saving as an attachment in an email message, or placing in a public folder. +Indicates that the intent is to share, or export, the object. +For example, saving as an attachment in an email message, or placing in a public folder. Sensitive information should probably not be preserved in this case. .It Dv XATTR_OPERATION_INTENT_SYNC Indicates that the intent is to sync the object to a service like iCloud Drive. .El .Sh FLAGS Various flags are defined by the type -.Dt xattr_flags_t ; -the currently-defined values for this are +.Vt xattr_flags_t ; +the currently-defined values for this are: .Bl -tag -width XATTR_FLAG_CONTENT_DEPENDENT .It Dv XATTR_FLAG_NO_EXPORT This indicates that the extended attribute should not be exported, or shared. @@ -94,15 +97,16 @@ This is used with .It Dv XATTR_FLAG_CONTENT_DEPENDENT This indicates that the extended attribute is tied to the contents of the file (or vice versa), such that it should be re-created when the contents -are changed. A checksum, for example, should not be copied, and would thus -be marked with this flag. +are changed. +A checksum, for example, should not be copied, and would thus be marked with this flag. .It Dv XATTR_FLAG_NEVER_PRESERVE This indicates that the extended attribute should never be copied from a source object to a destination, no matter what the given intent is. .It Dv XATTR_FLAG_SYNCABLE This indicates that the extended attribute should be copied when the file -is synced on services like iCloud Drive. Sync services may enforce additional -restrictions on the acceptable size and number of extended attributes. +is synced on services like iCloud Drive. +Sync services may enforce additional restrictions on the acceptable size and number +of extended attributes. .El .Sh EXAMPLE The following example is a simple function that, given an extended attribute