diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index a82701c4..4df062a3 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -27,3 +27,74 @@ __Some things that will increase the chance that your pull request is accepted__ * Follow our [coding style guide](../docs/CODE_OF_CONDUCT.md). * Write a good commit message. + +# Branch Naming Conventions +The structure and design conventions are based mainly on the “Conventional Branch” concept and the team’s current work process. + +## Purpose +1. Purpose-driven Branch Names: Each branch name clearly indicates its purpose, making it easy for all developers to understand what the branch is for. +2. Integration with CI/CD: By using consistent branch names, it can help automated systems (like Continuous Integration/Continuous Deployment pipelines) to trigger specific actions based on the branch type. +3. Better type and version control: Branches are related to a specific type of change in the code and the version that is being worked on, allowing for a much better backtracking. + +## Basic Rules +- Naming structure: +`/` +- Naming prefixes (Types): + - feature/: For new features (e.g., feature/add-login-page). + - fix/: For general fixes or code correcting. + - update/: For code upgrade or non-breaking code refactoring. + - chore/: For non-code tasks like dependency, docs updates (e.g., chore/update-dependencies). + - release/: For branches preparing a release (e.g., release/v1.2.0). +- Use Lowercase Alphanumeric and Hyphens: Always use lowercase letters (a-z), numbers (0-9), and hyphens to separate words. Avoid special characters, underscores, or spaces. +- No Consecutive or Trailing Hyphens: Ensure that hyphens are used singly, with no consecutive hyphens (feature/new--login) or at the end (feature/new-login-). +- Keep It Clear and Concise: The branch name should be descriptive yet concise, clearly indicating the purpose of the work. +- Include Ticket Numbers: If applicable, include the ticket number from your project management tool to make tracking easier. For example, for a ticket issue-123, the branch name could be feature/issue-123-new-login. + +## General Project Structure +``` +main +├── release/v1.x.x +│ ├── feature/lorem-ipsum +│ └── fix/ipsum-dolor +├── release/v2.x.x +│ └── chore/dolor-sit +└── release/v3.x.x + ├── update/sit-amet + ├── feature/amet-consectetur + └── chore/consectetur-adipiscing +``` +- main: This branch should only contain working versions completely tested and only receive pull requests from finalized release branches. All new release/ branches must come from this one. +- release: The creation of these branches should be done from the last version of main when created. Only this branches can make pull requests to main. +- Other branches: All working branches should come from a release branch. These are the only ones that should receive direct commits from changes made by development staff. These branches should only do PR to the active release branch. + +# Commits Naming Conventions +The structure and design conventions are based mainly on the “Conventional Branch” concept and the team’s current work process. + +## Purpose +1. Automatically generating CHANGELOGs: When the commits are concise, and precise changelogs can be automated or semi-automated from the commit history. +2. Communicating the nature of changes to everyone: Being able to understand the type of changing by just looking at a name and understanding the basics of a commit just by the title is useful in analysis and backtrack situations. +3. Building code habits and good practices: Maintaining structured commits also leads to more precise coding practices and coherent code changes. + +## Basic Rules +- Naming structure: +``` +: + +[optional body] +``` +- Naming prefixes (Types): + - fix: A commit of the type fix patches a bug in your codebase. + - feat: A commit of the type feat introduces a new feature to the codebase + - !: A commit that appends a ! after the type introduces a breaking change. A breaking change can be part of commits of any type. + - update: A commit of type update changes existing code or refactors functions without changing functionality of adding features in your codebase. + - build: A commit of type build changes build process in your codebase. + - docs: A commit of type docs is for creation or updating documentation in your codebase. + - chore: A commit of type chore is for every other minor task in your codebase. +- Commits must be prefixed with a type, which consists of a noun, feat, fix, etc., followed by the optional !, and required terminal colon and space. +- A description must immediately follow the colon and space after the type prefix. The description is a short summary of the code changes, e.g., fix: array parsing issue when multiple spaces were contained in string. +- A longer commit body may be provided after the short description, providing additional contextual information about the code changes. The body must begin one blank line after the description. +- A commit body is free-form and may consist of any number of newline separated paragraphs. + +# Bibliography +[1] “Conventional Branch,” Conventional Branch, 2025. https://conventional-branch.github.io/ (accessed 2025).
+[2] “Conventional Commits,” Conventional Commits. https://www.conventionalcommits.org/en/v1.0.0/ (accessed 2025). diff --git a/.github/ISSUE_TEMPLATE/hba_report.md b/.github/ISSUE_TEMPLATE/hba_report.md new file mode 100644 index 00000000..0884bfb3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/hba_report.md @@ -0,0 +1,15 @@ +--- +name: HBA report +about: Report a HBA that is working on the specific enviroment +assignees: piste-jp-ibm +lebels: HBA Report +title: HBA list request + +--- + +**Fill in the informatio of HBA and OS** + +| HBA type | Vendor | I/F type (FC, SAS etc) | OS | LTFS version or commit hash | Note | +| -------- | ------ | ---------------------- | --- | --------------------------- | ---- | +| | | | | | | + diff --git a/README.md b/README.md index 0fa2a247..9b1d28c9 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,13 @@ ![GH Action status](https://github.com/LinearTapeFileSystem/ltfs/workflows/CentOS7%20Build%20Job/badge.svg?branch=master) [![BSD License](http://img.shields.io/badge/license-BSD-blue.svg?style=flat)](LICENSE) -# Linear Tape File System (LTFS) +# About this branch -Linear Tape File System (LTFS) is a filesystem to mount a LTFS formatted tape in a tape drive. Once LTFS mounts a LTFS formatted tape as filesystem, user can access to the tape via filesystem API. +This is the `master` branch of the LTFS project. At this time, this branch is used for version 2.5 development. So it wouldn't be stable a little. Please consider to follow the tree on `v2.4-stable` branch if you want to use stable codes. + +# What is the Linear Tape File System (LTFS) + +The Linear Tape File System (LTFS) is a filesystem to mount a LTFS formatted tape in a tape drive. Once LTFS mounts a LTFS formatted tape as filesystem, user can access to the tape via filesystem API. Objective of this project is being the reference implementation of the LTFS format Specifications in [SNIA](https://www.snia.org/tech_activities/standards/curr_standards/ltfs). @@ -103,9 +107,91 @@ These instructions will get you a copy of the project up and running on your loc | Quantum | LTO8 (Only Half Height) | T.B.D. | | Quantum | LTO9 (Only Half Height) | T.B.D. | -## Installing +## LTFS Format Specifications + +LTFS Format Specification is specified data placement, shape of index and names of extended attributes for LTFS. This specification is defined in [SNIA](https://www.snia.org/tech_activities/standards/curr_standards/ltfs) first and then it is forwarded to [ISO](https://www.iso.org/home.html) as ISO/IEC 20919 from version 2.2. + +The table below show status of the LTFS format Specification + + | Version | Status of SNIA | Status of ISO | + |:-------:|:---------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------:| + | 2.2 | [Published](http://snia.org/sites/default/files/LTFS_Format_2.2.0_Technical_Position.pdf) | [Published as `20919:2016`](https://www.iso.org/standard/69458.html) | + | 2.3.1 | [Published](https://www.snia.org/sites/default/files/technical_work/LTFS/LTFS_Format_2.3.1_TechPosition.PDF) | - | + | 2.4 | [Published](https://www.snia.org/sites/default/files/technical_work/LTFS/LTFS_Format_2.4.0_TechPosition.pdf) | - | + | 2.5.1 | [Published](https://www.snia.org/sites/default/files/technical-work/ltfs/release/SNIA-LTFS-Format-2-5-1-Standard.pdf) | [Published as `20919:2021`](https://www.iso.org/standard/80598.html) | + +# How to use the LTFS (Quick start) + +This section is for a person who already has a machine with the LTFS installed. Instructions on how to use the LTFS is also available on [Wiki](https://github.com/LinearTapeFileSystem/ltfs/wiki). + +## Step1: List tape drives + +`# ltfs -o device_list` + +The output is as follows. You have 3 drives in this example and you can use "Device Name" field, like `/dev/sg43` in this case, as the argument of ltfs command to mount the tape drive. + +``` +50c4 LTFS14000I LTFS starting, LTFS version 2.4.0.0 (10022), log level 2. +50c4 LTFS14058I LTFS Format Specification version 2.4.0. +50c4 LTFS14104I Launched by "/home/piste/ltfsoss/bin/ltfs -o device_list". +50c4 LTFS14105I This binary is built for Linux (x86_64). +50c4 LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11). +50c4 LTFS17087I Kernel version: Linux version 3.10.0-514.10.2.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 i386. +50c4 LTFS17089I Distribution: NAME="Red Hat Enterprise Linux Server". +50c4 LTFS17089I Distribution: Red Hat Enterprise Linux Server release 7.3 (Maipo). +50c4 LTFS17089I Distribution: Red Hat Enterprise Linux Server release 7.3 (Maipo). +50c4 LTFS17085I Plugin: Loading "sg" tape backend. +Tape Device list:. +Device Name = /dev/sg43, Vender ID = IBM , Product ID = ULTRIUM-TD5 , Serial Number = 9A700L0077, Product Name = [ULTRIUM-TD5] . +Device Name = /dev/sg38, Vender ID = IBM , Product ID = ULT3580-TD6 , Serial Number = 00013B0119, Product Name = [ULT3580-TD6] . +Device Name = /dev/sg37, Vender ID = IBM , Product ID = ULT3580-TD7 , Serial Number = 00078D00C2, Product Name = [ULT3580-TD7] . +``` + +## Step2: Format a tape + +As described in the LTFS format specifications, LTFS uses the partition feature of the tape drive. This means you can't use a tape just after you purchase a tape. You need format the tape before using it on LTFS. + +To format a tape, you can use `mkltfs` command like + +`# mkltfs -d 9A700L0077` + +In this case, `mkltfs` tries to format a tape in the tape drive `9A700L0077`. You can use the device name `/dev/sg43` instead. + +## Step3: Mount a tape through a tape drive + +After you prepared a formatted tape, you can mount it through a tape drive like + +`# ltfs -o devname=9A700L0077 /ltfs` + +In this command, the ltfs command will try to mount the tape in the tape drive `9A700L0077` to `/ltfs` directory. Of course, you can use a device name `/dev/sg43` instead. + +If the mount process is successfully done, you can access to the LTFS tape through `/ltfs` directory. + +You must not touch any `st` devices while ltfs is mounting a tape. + +## Step4: Unmount the tape drive + +You can use following command when you want to unmount the tape. The ltfs command try to write the current meta-data to the tape and close the tape cleanly. + +`# umount /ltfs` + +One thing you need to pay attention to here is, that the unmount command continues to work in the background after it returns. It just initiates a trigger to notify the the ltfs command of the unmount request. Actual unmount is completed when the ltfs command is finished. + +## The `ltfsee_ordered_copy` utility + +The [`ltfsee_ordered_copy`](https://github.com/LinearTapeFileSystem/ltfs/wiki/ltfs_ordered_copy) is a program to copy files from source to destination with LTFS order optimization. + +It is written in python and it can work with both python2 and python3 (Python 2.7 or later is strongly recommended). You need to install the `pyxattr` module for both python2 and python3. + +# Building the LTFS from this GitHub project + +These instructions will get a copy of the project up and running on your local machine for development and testing purposes. + +## Prerequisites for build + +Please refer [this page](https://github.com/LinearTapeFileSystem/ltfs/wiki/Build-Environments). -### Build and install on Linux +## Build and install on Linux ``` ./autogen.sh @@ -120,7 +206,7 @@ In some systems, you might need `sudo ldconfig -v` after `make install` to load #### Parameter settings of the sg driver -LTFS uses the sg driver by default. You can improve reliability to change parameters of the sg driver below. +### Buildable Linux distributions ``` def_reserved_size=1048576 @@ -217,9 +303,9 @@ Currently, automatic build checking is working on GitHub Actions and Travis CI. For Ubuntu20.04 and Debian10, dummy `icu-config` is needed in the build machine. See Issue [#153](https://github.com/LinearTapeFileSystem/ltfs/issues/153). -### Build and install on OSX (macOS) +## Build and install on OSX (macOS) -#### Recent Homedrew system setup +### Recent Homebrew system setup Before build on macOS, you need to configure the environment like below. @@ -230,7 +316,7 @@ export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/libxml export PATH="$PATH:$ICU_PATH:$LIBXML2_PATH" ``` -#### Old Homedrew system setup +### Old Homebrew system setup Before build on OSX (macOS), some include path adjustment is required. ``` @@ -238,7 +324,7 @@ brew link --force icu4c brew link --force libxml2 ``` -#### Building LTFS +### Building LTFS On OSX (macOS), snmp cannot be supported, you need to disable it on configure script. And may be, you need to specify LDFLAGS while running configure script to link some required frameworks, CoreFundation and IOKit. ``` @@ -250,13 +336,13 @@ make install `./configure --help` shows various options for build and install. -#### Buildable systems +#### Buildable macOS systems | OS | Xcode | Package system | Status | |:-: |:-: |:-: |:-: | | macOS 10.14.6 | 11.3 | Homebrew | Probably OK | -### Build and install on FreeBSD +## Build and install on FreeBSD Note that on FreeBSD, the usual 3rd party man directory is /usr/local/man. Configure defaults to using /usr/local/share/man. So, override it on the command line to avoid having man pages put in the wrong place. @@ -269,9 +355,10 @@ make install #### Buildable versions - | Version | Arch | Status | - |:-: |:-: |:-: | - | 11 | x86_64 | OK | + | Version | Arch | Status | + |:-------:|:-------:|:------------------------------:| + | 11 | x86\_64 | OK - Not checked automatically | + | 12 | x86\_64 | OK - Not checked automatically | ### Build and install on NetBSD @@ -284,11 +371,11 @@ make install #### Buildable versions - | Version | Arch | Status | - |:-: |:-: |:-: | - | 8.1 | amd64 | OK | - | 8.0 | i386 | OK | - | 7.2 | amd64 | OK | + | Version | Arch | Status | + |:-------:|:-----:|:------------------------------:| + | 8.1 | amd64 | OK - Not checked automatically | + | 8.0 | i386 | OK - Not checked automatically | + | 7.2 | amd64 | OK - Not checked automatically | ## Contributing diff --git a/autogen.sh b/autogen.sh index 504a9748..113fb3f2 100755 --- a/autogen.sh +++ b/autogen.sh @@ -46,6 +46,6 @@ else libtoolize fi -autoconf -autoheader -automake --add-missing --copy +autoconf || echo "Ignore warning" +autoheader || echo "Ignore warning" +automake --add-missing --copy || echo "Ignore warning" diff --git a/configure.ac b/configure.ac index f28d2b28..faed022e 100644 --- a/configure.ac +++ b/configure.ac @@ -49,7 +49,7 @@ AC_USE_SYSTEM_EXTENSIONS AC_PROG_CC_C99 AM_PROG_CC_C_O AM_PROG_AR -AC_PROG_LIBTOOL +LT_INIT dnl dnl Detecting OS @@ -152,22 +152,13 @@ AC_ARG_ENABLE([snmp], AC_MSG_RESULT([$snmp]) dnl -dnl Handle --enable-new-locking (default: depend on platform) -dnl NetBSD: old locking is broken, force new locking -dnl others: default to old locking +dnl Handle --enable-new-locking (default: yes) dnl -if test "x${host_netbsd}" = "xyes" -then - use_new_locking_default=yes -else - use_new_locking_default=no -fi - AC_MSG_CHECKING([whether to enable new locking system or not]) AC_ARG_ENABLE([new-locking], [AS_HELP_STRING([--enable-new-locking],[Use new locking system or not])], [use_new_locking=$enableval], - [use_new_locking=$use_new_locking_default] + [use_new_locking=yes] ) AC_MSG_RESULT([$use_new_locking]) @@ -193,6 +184,28 @@ AC_ARG_ENABLE([buggy_ifs], ) AC_MSG_RESULT([$buggy_ifs]) +dnl +dnl Handle --enable-xml-indent (default:no) +dnl +AC_MSG_CHECKING([whether to enable xml indentation for index]) +AC_ARG_ENABLE([xml_indent], + [AS_HELP_STRING([--enable-xml-indent],[Enable XML indentation for index])], + [xml_indent=$enableval], + [xml_indent=no] +) +AC_MSG_RESULT([$xml_indent]) + +dnl +dnl Handle --enable-format-spec25 (default:no) +dnl +AC_MSG_CHECKING([whether to enable format spec 2.5 support or not]) +AC_ARG_ENABLE([format_spec25], + [AS_HELP_STRING([--enable-format-spec25],[Support format spec 2.5])], + [format_spec25=$enableval], + [format_spec25=no] +) +AC_MSG_RESULT([$format_spec25]) + dnl dnl Handle extra compile flags for tape driver dnl @@ -337,16 +350,20 @@ fi dnl dnl Check for ICU dnl -ICU_MODULE_CFLAGS="`icu-config --cppflags 2> /dev/null`"; -ICU_MODULE_LIBS="`icu-config --ldflags 2> /dev/null`"; -if test -z "$ICU_MODULE_LIBS" +AC_MSG_CHECKING(ICU version) +ICU_MODULE_VERSION="`pkg-config --modversion icu-i18n 2> /dev/null`"; +ICU_MODULE_CFLAGS="`pkg-config --cflags icu-i18n 2> /dev/null`"; +ICU_MODULE_LIBS="`pkg-config --libs icu-i18n 2> /dev/null`"; +AC_MSG_RESULT([$ICU_MODULE_VERSION]) + +if test -z "$ICU_MODULE_VERSION" then - PKG_CHECK_MODULES([ICU_MODULE], [icu >= 0.21]) + PKG_CHECK_MODULES([ICU_MODULE], [icu-uc >= 0.21]) fi -AC_MSG_CHECKING([use latest ICU]) +AC_MSG_CHECKING([for using ICU6x (unorm2) functions forcibly]) AC_ARG_ENABLE([icu_6x], - [AS_HELP_STRING([--enable-icu-6x],[Support handling of ICU 6x functions])], + [AS_HELP_STRING([--enable-icu-6x],[Force to use ICU6x (unorm2) functions])], [icu_6x=$enableval], [icu_6x=no] ) @@ -354,13 +371,16 @@ AC_MSG_RESULT([$icu_6x]) if test "x${icu_6x}" = "xyes" then - AC_MSG_CHECKING(for ICU version) - ICU_MODULE_VERSION="`icu-config --version 2> /dev/null`"; - if test "${ICU_MODULE_VERSION%%.*}" -ge "60" + AM_EXTRA_CPPFLAGS="${AM_EXTRA_CPPFLAGS} -D USE_UNORM2" +else + AC_MSG_CHECKING(for using unorm2 functions) + if test "${ICU_MODULE_VERSION%%.*}" -ge "56" then - AM_EXTRA_CPPFLAGS="${AM_EXTRA_CPPFLAGS} -D ICU6x" + AM_EXTRA_CPPFLAGS="${AM_EXTRA_CPPFLAGS} -D USE_UNORM2" + AC_MSG_RESULT(yes) + else + AC_MSG_RESULT(no) fi - AC_MSG_RESULT([$ICU_MODULE_VERSION]) fi dnl @@ -374,7 +394,7 @@ if test "x${snmp}" != "xno" then SNMP_ENABLE="-D ENABLE_SNMP" SNMP_MODULE_CFLAGS="`net-snmp-config --cflags 2> /dev/null`"; - SNMP_MODULE_LIBS_A="`net-snmp-config --agent-libs` 2> /dev/null"; + SNMP_MODULE_LIBS_A="`net-snmp-config --agent-libs 2> /dev/null`"; SNMP_MODULE_LIBS="`net-snmp-config --libs 2> /dev/null`"; if test -z "$SNMP_MODULE_LIBS_A" then @@ -429,11 +449,12 @@ SAVE_CFLAGS=$CFLAGS CFLAGS="$CFLAGS ${LIBXML2_MODULE_CFLAGS}" AC_MSG_CHECKING(for XML_PARSE_HUGE) AC_CACHE_VAL(ac_cv_xml_huge, - AC_TRY_COMPILE([#include ], - [int foo = XML_PARSE_HUGE; return 0], - ac_cv_xml_huge=yes, - ac_cv_xml_huge=no - )) + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM([#include ], + [int foo = XML_PARSE_HUGE; return 0;])], + [ac_cv_xml_huge=yes], + [ac_cv_xml_huge=no] + )) AC_MSG_RESULT($ac_cv_xml_huge) CFLAGS="$SAVE_CFLAGS" @@ -500,6 +521,16 @@ then AM_CPPFLAGS="${AM_CPPFLAGS} -DPOSIXLINK_ONLY" fi +if test "x$xml_indent" = "xyes" +then + AM_CPPFLAGS="${AM_CPPFLAGS} -DINDENT_INDEXES" +fi + +if test "x$format_spec25" = "xyes" +then + AM_CPPFLAGS="${AM_CPPFLAGS} -DFORMAT_SPEC25" +fi + dnl dnl Specify CPU specific optimizer options for CRC calculation dnl @@ -573,6 +604,7 @@ fi if test "x${host_mac}" = "xno" then MODULE_CHECK_LD='-Wl,--no-undefined,--as-needed' + AM_LDFLAGS="${AM_LDFLAGS} -ldl" fi AM_CFLAGS=`echo ${AM_CFLAGS} | ${SED} 's|-D_FORTIFY_SOURCE=. ||g'` diff --git a/contrib/fssim/README.md b/contrib/fssim/README.md new file mode 100644 index 00000000..c4550c18 --- /dev/null +++ b/contrib/fssim/README.md @@ -0,0 +1,10 @@ +# LTFS File System Simulator (fssim) + +The major change in version 2.5 of the [LTFS Format Specification](https://www.snia.org/sites/default/files/technical_work/LTFS/LTFS_Format_v2.5_Technical_Position.pdf) is the addition of Incremental Indexes. Incremental Indexes allow an LTFS index written to the Data Partition of the tape to contain only the changes from the prior index, rather than the full state of the file system. For a full discussion of Incremental Indexes, see Appendix H of the Format Specification referenced above. + +When we first proposed Incremental Indexes there was considerable discussion concerning what they would need to contain, and how they would be implemented. I decided to write a simulator that would mimic file system operations, and create full and incremental XML indexes very similar to those in LTFS as a Proof of Concept. This simulator also includes programs to merge full and incremental directories created by it and verify the results, as well as a directory of test cases built by various members of the LTFS project. + +The programs are meant to demonstrate how Incremental Indexes work, and also to potentially serve as an example of how they could be implemented. + +Full documentation of the programs is contained in the README file in the docs directory. + diff --git a/contrib/fssim/docs/README.md b/contrib/fssim/docs/README.md new file mode 100644 index 00000000..06300c54 --- /dev/null +++ b/contrib/fssim/docs/README.md @@ -0,0 +1,381 @@ + +# LTFS File System Simulator (fssim) +# Usage Guide and Examples + + +## Overview + +The File System Simulator programs are used to simulate file system activity, +create full and incrmental indexes reflecting that activity, update a full +index using incremental indexes, and verify the correctness of that update. + +The first program in the set is _fssim_. It simulates file system activity such +as creating or deleting files and directories, moving and copying files or +directories, listing information, etc. It also writes full and incremental +XML indexes to text files on request. + +The second program is _fsidxupd_ (perhaps a poor choice of names). It updates +a full index created by fssim with one or more incremental indexes created +sequentially in the same execution of fssim, then creates a full index +from the result. The resulting full index should exactly match the +corresponding full index from fssim. + +The programs were written by David Pease at IBM's Almaden Research Center in +May-August of 2018, and significantly revised and improved by the original +author (now retired) in September of 2020. + +## Invoking the programs: + +The programs are python executables, meaning that they can be invoked from +the command line in Unix-like environments (such as Linux and Mac) as long as +the executable bit is on and /usr/bin/python points to the python interpreter. The programs are written so that they run correctly in either python 2 or +python 3. + +In Windows, it should be possible to invoke them by a command such as "python +fsXXX" or by renaming the two program files to fsXXX.py and invoking them by +name (if python is properly configured). + +There are two additional files besides the _fssim_ and _fsidxupd_ programs. They +are _fscommon.py_ and _fsglobals.py_. Both of these are imported by the two +command programs; all four files should reside in the same directory. + +### fssim + +The format of the _fssim_ command is: + + fssim [-d] [command file] + +where: +* -d: prints a ton of debugging messages (probably meaningfull only to me) during processing +* command file: executes commands from the specified text file at startup + + A command file is simply a text file containing commands as they would be + entered from the keyboard; blank lines and comment lines starting with "#" + are allowed. Command files make it easy to design and debug a scenario, + then execute it multiple times (often while modifying code + in-between). After a command file is executed (without encountering an + "exit" command), the program drops into its normal keyboard input mode + (which is terminated by entering "exit"). + +### fsidxupd + +The format of the _fsidxupd_ command is: + + fsidxupd [-d] [ ...] + +where: +* -d: prints lots of debugging messages during processing +* fullindex: a file containing a full index from fssim +* incrindexes:a list of files containing incremental indexes from fssim + +The first file name must identify a full index created by _fssim_. The second +and subsequent names, if present, must identify incremental indexes created +sequentially by _fssim_. The output of the program (currently) has a hard-coded +file name of _upd-full.xml_. + +## fssim Commands + +The _fssim_ commands are closely modeled after the Unix shell commands that +perform the same function; however, they are often greatly simplified versions +(the goal was to implement a simulator, not re-write bash or busybox). + +Unrecognized commands or commands with an invalid number of operands will +return an **"\*\* Invalid"** message. Commands that do not execute successfully +(for any reason) will return the message **"\*\* Failed"**. That is the extent +of error messages. + +The command prompt indicates the current working directory (as it typically +does in bash), and starts at root ("/"). File and directory names can be +entered as full paths or relative to the current directory; however, the only +place that ".." is recognized is as the single operand of a _cd_ command, and +the only place that "." is recognized is as the single target for a _mv_ or _cp_ command. The directory separator character is "/". The command prompt +environment supports command recall and editing using the _readline_ library (tested on Linux and Mac). + +The command processor respects strings enclosed in double quotes ("). Quoted +strings can be used to create object names with embedded blanks or +to echo strings with arbitrary blank spaces. For example: + + echo Hello > "file 1"` + echo " world!" >> "file 1"` + cat "file 1"` + +The _help_ command will print a short description of all of the commands. +Here is a slightly more detailed command summary (square brackets indicate +optional parameters, curly braces indicate a choice): + +|name|operands|description| +|:-|:-|:-| +|cat|name|write a file's contents to the screen| +|cd|[dirname]|change current directory; with no operands, changes to root| +|cls||clear the screen (tested in Linux terminal window)| +|cp|[-r] name name|copy a file from one name to another; with -r option, copy directory contents recursively| +|dir|[-a] [name]|an alias for ls| +|echo|data [{>\|>>} name]|write data to the screen or to a file| +|fsck||verify that all object flags have been correctly reset | +|index|[{-i\|-f}] fileprefix|write full and/or incremental indexes to files using names prefixed with the specified prefix (see below)| +|ll|[name]|list long: an alias for ls -a | +|log|[-t]|list current update log entries (optionally in time order)| +|ls|[-a] [name]|list file or directory info (see more info below)| +|md|dirname [...]|an alias for mkdir| +|mkdir|dirname[...]|create directories or full directory paths| +|mv|name name|move/rename a file or directory| +|rd|dirname|an alias for rmdir| +|rm|filename|remove a file| +|rmdir|dirname|remove a directory (if it is empty)| +|touch|name [...] |update the timestamp of existing files or directories, or create new files (with directory paths if needed)| +|tree|[-o]|show a tree view of the entire file system; with -o option, also include oids| + +The _ls_ (or _dir_) command prefixes each file or directory with some attributes in +the order **"dmn"**. A **"d"** in the first position indicates a directory. An **"m"** +in the second position indicates that the file or directory has been modified +since the last index was written. An **"n"** in the third position indicates that +the directory is new since the last index (and thus must be written in its entirety with all of its children to an incremental index). Following the +attributes are the object id, the modification time, and the object size. For +files, the size is the number of data bytes in the file; for directories, it +is the number of children in the directory. When the object of the _ls_ +command is a directory, the "-a" (all) option will also show an entry for "." +which lists the status of the directory itself. (The _ll_ command is shorthand +for _ls -a_.) + +The _tree_ command shows a full file system tree view; using the "-o" option +shows the object ids (oids) following the object names, as in this example: + + / (0) + |-file1 (1) + |=dir1 (2) + |=dir2 (4) + |-file2 (5) + |=dir3 (3) + +As can be seen from the example, a "-" in front of a name in the tree view +indicates a file, while "=" indicates a directory. Within a directory +files are listed first followed by subdirectories, in alphabetical order. + +The _mkdir_ (or _md_), _touch_, and _echo_ commands will create any missing lower-level +directories required in the directory or file path. + +A note using on the _cp_ command with the "-r" option (recursive copy). This +command option requires two existing directory names as input; it copies the +**contents of the first directory to the second, recursively. Thus, using a +directory structure such as that shown in the output of _tree_ above, the +following command: + + cp -r dir1/dir2 dir3 + +would copy only file2 to dir3. In order to end up with dir2 and file2 under +dir3, one could use either + + cp -r dir1 dir3 + +or + + md dir3/dir2 + cp -r dir1/dir2 dir3/dir2 + +Note, however, that the first option will copy everything under dir1 to dir3; +if dir1 had files or directories other than dir2 under it, it would copy +everything and would not give the desired results. + +## Indexes + +By default, the _index_ command writes both a full index and an increment index +to a pair of files whose names start with the specified prefix. For example, +the command _index t0_ will create files _t0-full.xml_ and _t0-incr.xml_ in +the user's current directory. (The rationale for writing both indices is that +that one can start from any full index written, apply any number of subsequent +incremental indexes they wish, and the result should be the same as the full +index corresponding to the last incremental index used.) The _index_ command +allow overriding the default behavior by using the "-i" (incremental index +only) or "-f" (full index only) options. To simulate normal system operation, +(and minimize the number of files produced during testing), one could specify +"-f" for the index produced at the start of the run (e.g., _index -f t0_), then +use "-i" for all subsequent indexes up to the last one, and use neither flag +for the final index. This would give a starting and ending full index, and a +full set of incremental indexes to validate against the final full index. + +When testing indexes it's probably a good idea to start a command file with +the command _index [-f] t0_ (or the prefix of your choice); this creates a +starting point for incremental indexes from the beginning of the session. _index_ commands can be interspersed in a command stream at any point, and the +resulting full and incremental indexes can be inspected (and processed using _fsidxupd_). + +The _fsck_ command searches the entire file system for any files or directories +with **New** or **Modified** flags. This command can be used after an _index_ command with the +"-i" (incremental only) flag to verify that all objects with these flags have +been visited, and that all such flags have been correctly reset. + +A pair of "t0" indexes (created at startup) looks like this: + + + 0 + + + + + + + 0 + + + + + +The two are essentially identical, since the root directory is the only one in +existence, and it is new and thus shows up in the incremental index as well. (Actually, the t0 incremental index is useless, but it helps illustrate how +indexes work.) + +Running _fssim_ with this set of commands (e.g., in a command file): + + index t0 + mkdir Dir1 + touch Dir2/Dir3/foo + touch Dir2/Dir3/bar + index t1 + touch Dir2/Dir3/baz + rmdir Dir1 + index t2 + +produced the following two "t2" indexes: + + + 0 + + + + Dir2 + 2 + + + + Dir3 + 3 + + + + 5 + bar + + + + 6 + baz + + + + 4 + foo + + + + + + + + + + + + + + Dir1 + + + + Dir2 + + + Dir3 + + + + baz + 6 + + + + + + + + + +## Using fsidxupd + +_fsidxupd_ is used to verify the correctness of incremental indexes and their +application to a full index. For example, using the sample list of commands +in the Index section above, three pairs of indexes will be created by fssim +when running the commands; the file names will be _t0-full.xml, t0-incr.xml, +t1-full.xml, t1-incr.xml, t2-full.xml_, and _t2-incr.xml_. The following +_fsidxupd_ command will take the starting full index and apply the two appropriate incremental indexes: + + fsidxupd t0-full.xml t1-incr.xml t2-incr.xml + +The output of this command will be a file named _upd-full.xml_. In this example, +if the system is working correctly _upd-full.xml_ will be identical to +_t2-full.xml_. In Unix-like systems, the command: + + diff upd-full.xml t2-full.xml + +would show no output (no differences). + +## Program Organization + +The file _fscommon.py_ contains all of the file system logic. It contains +object definitions for directory and file objects, and implements all of +the low-level logic for managing those objects. (In function, it roughly +corresponds to the vfs module in a Unix-like OS.) It also contains the +the python version-agnostic print function (_fsprt_). + +The file _fsglobals.py_ defines global variables that must be shared between +the different modules. In order to provide a consistent naming reference +to these shared global variables, they must be in a separate python module +and that module must be imported with the same name in each program. (For +succinctness, I import it as "g", so that global values are referenced as +_g.variablename_.) + +The file _fssim_ is the command processor for user file system commands; it +processes commands from a command file and/or the command line. (In Unix +terms, it implements the user shell and shell commands.) + +The file _fsidxupd_ is a utility program that reads, updates, and writes +indexes. It uses file system functions from _fscommon.py_ to update a full +index using data from incremental indexes. + +## fsruntest + +The programs also include a bash shell script to automate testing. The +format of the command is: + + fsruntest [-d] + +The command file is used to run the test. It may end with an _exit_ command, +or the _exit_ command can be entered manually while the test is running. The +"-d" option runs _fssim_ and _fsidxupd_ with debugging messages enabled. + +The script first invokes _fssim_, passing in the name of the command file. It +then invokes _fsidxupd_ to apply all of the subsequent incremental indexes to +the first full index file created. Finally, it compares the output of _fsidxupd_ +to the final full index created by _fssim_. If the output of diff is not empty, +the output is displayed, preceded by a highlighted error line (so that it stands out or +can be searched for in a large test run). + +Since it is a bash shell script, the test script runs on both Linux and Mac +systems, and with the proper Linux subsystem configured perhaps even on +Windows 10 (or earlier, using Cygwin). + + +## Final Comments + +The _oids_ in the files and directories correspond to the _fileuid_ in LTFS. +Initially I did not believe that _oids_ (and thus _fileuids_) were important +to processing incremental indexes. It turns out that I was wrong, and that +a unique identifier for an object is important to the correct application +of an incremental index in one important case. Timestamps are the last +object modification time. I think all other object fields are obvious. + +The programs have not been intensively debugged. There may be lingering +errors in the commands and/or in index generation. It all seems to work, +but if you find errors (or even think you **may** have found an error) please +let me know. The better debugged the programs are, the more reliable our +results will be. + +#### - David Pease - pease@coati.com diff --git a/contrib/fssim/docs/README.txt b/contrib/fssim/docs/README.txt new file mode 100644 index 00000000..f1918670 --- /dev/null +++ b/contrib/fssim/docs/README.txt @@ -0,0 +1,393 @@ + +LTFS File System Simulator (fssim) Usage Guide and Examples (README) + +Overview +======== + +The File System Simulator programs are used to simulate file system activity, +create full and incrmental indexes reflecting that activity, update a full +index using incremental indexes, and verify the correctness of that update. + +The first program in the set is fssim. It simulates file system activity such +as creating or deleting files and directories, moving and copying files or +directories, listing information, etc. It also writes full and incremental +XML indexes to text files on request. + +The second program is fsidxupd (perhaps a poor choice of names). It updates +a full index created by fssim with one or more incremental indexes created +sequentially in the same execution of fssim, then creates a full index +from the result. The resulting full index should exactly match the +corresponding full index from fssim. + +The programs were written by David Pease at IBM's Almaden Research Center in +May-August of 2018, and significantly revised and improved by the original +author (now retired) in September of 2020. + +Invoking the programs: +====================== + +The programs are python executables, meaning that they can be invoked from +the command line in Unix-like environments (such as Linux and Mac) as long as +the executable bit is on and /usr/bin/python points to the python interpreter. +The programs are written so that they run correctly in either python 2 or +python 3. + +In Windows, it should be possible to invoke them by a command such as "python +fsXXX" or by renaming the two program files to fsXXX.py and invoking them by +name (if python is properly configured). + +There are two additional files besides the fssim and fsidxupd programs. They +are fscommon.py and fsglobals.py. Both of these are imported by the two +command programs; all four files should reside in the same directory. + +The format of the fssim command is: + + fssim [-d] [command file] + +where: + -d: prints a ton of debugging messages (probably meaningfull + only to me) during processing + command file: executes commands from the specified text file at startup + + A command file is simply a text file containing commands as they would be + entered from the keyboard; blank lines and comment lines starting with "#" + are allowed. Command files make it easy to design and debug a scenario, + then execute it multiple times (often while modifying code in-between). + After a command file is executed (without encountering an "exit" command), + the program drops into its normal keyboard input mode (which is terminated + by entering "exit"). + +The format of the fsidxupd command is: + + fsidxupd [-d] [ ...] + +where: + -d: prints lots of debugging messages during processing + fullindex: a file containing a full index from fssim + incrindexes: a list of files containing incremental indexes from fssim + +The first file name must identify a full index created by fssim. The second +and subsequent names, if present, must identify incremental indexes created +sequentially by fssim. The output of the program (currently) has a hard-coded +file name of upd-full.xml + +fssim Commands +============== + +The fssim commands are closely modeled after the Unix shell commands that +perform the same function; however, they are often greatly simplified versions +(the goal was to implement a simulator, not re-write bash). + +Unrecognized commands or commands with an invalid number of operands will +return a "** Invalid" message. Commands that do not execute successfully +(for any reason) will return the message "** Failed". That is the extent of +error messages. (This is only a test program.) + +The command prompt indicates the current working directory (as it typically +does in bash), and starts at root ("/"). File and directory names can be +entered as full paths or relative to the current directory; however, the only +place that ".." is recognized is as the single operand of a "cd" command, and +the only place that "." is recognized is as the single target for a mv or cp +command. The directory separator character is "/". The command prompt +environment supports command recall and editing (has been tested on Linux and +Mac). + +The command processor respects strings enclosed in double quotes ("). +Quoted strings can be used to create object names with embedded blanks or +to echo strings with arbitrary blank spaces. For example: + +echo Hello > "file 1" +echo " world!" >> "file 1" +cat "file 1" + +The "help" command will print a short description of all of the commands. +Here is a slightly more detailed command summary (square brackets indicate +optional parameters, curly braces indicate a choice): + +cat name - write a file's contents to the screen +cd [dirname] - change current directory; with no operands, changes to root +cls - clear the screen (tested in Linux terminal window) +cp [-r] name name - copy a file from one name to another; with -r option, + copy directory contents recursively +dir [-a] [name] - an alias for ls +echo data [{>|>>} name] - write data to the screen or to a file +fsck - verify that all object flags have been correctly reset +index [{-i|-f}] fileprefix - write full and/or incremental indexes to files + using names prefixed with the specified prefix (see below) +ll [name] - list long: an alias for ls -a +log [-t] - list current update log entries (optionally in time order) +ls [-a] [name] - list file or directory info (see more info below) +md dirname [...] - an alias for mkdir +mkdir dirname [...] - create directories or full directory paths +mv name name - move/rename a file or directory +rd dirname - an alias for rmdir +rm filename - remove a file +rmdir dirname - remove a directory (if it is empty) +touch name [...] - update the timestamp of existing files or directories, + or create new files (with directory paths if needed) +tree [-o] - show a tree view of the entire file system; with -o + option, also include oids + +The ls (dir) command prefixes each file or directory with some attributes in +the order "dmn". A "d" in the first position indicates a directory. An "m" +in the second position indicates that the file or directory has been modified +since the last index was written. An "n" in the third position indicates that +the directory is new since the last index (and thus must be written in its +entirety with all of its children to an incremental index). Following the +attributes are the object id, the modification time, and the object size. For +files, the size is the number of data bytes in the file; for directories, it +is the number of children in the directory. When the object of the "ls" +command is a directory, the "-a" (all) option will also show an entry (".") +which lists the status of the directory itself. (The ll command is shorthand +for "ls -a".) + +The tree command shows a full file system tree view; using the "-o" option +shows the object ids (oids) following the object names, as in this example: + + / (0) + |-file1 (1) + |=dir1 (2) + |=dir2 (4) + |-file2 (5) + |=dir3 (3) + +As can be seen from the example, a "-" in front of a name in the tree view +indicates a file, while a "=" indicates a directory. Within a directory +files are listed first followed by subdirectories, in alphabetical order. + +The mkdir (md), touch, and echo commands will create any missing lower-level +directories required in the directory or file path. + +A note using on the cp command with the "-r" option (recursive copy). This +command option requires two existing directory names as input; it copies the +*contents* of the first directory to the second, recursively. Thus, using a +directory structure such as that shown in the output of "tree" above, the +following command: + + cp -r dir1/dir2 dir3 + +would copy only file2 to dir3. In order to end up with dir2 and file2 under +dir3, one could use either + + cp -r dir1 dir3 + +or + md dir3/dir2 + cp -r dir1/dir2 dir3/dir2 + +Note, however, that the first option will copy everything under dir1 to dir3; +if dir1 had files or directories other than dir2 under it, it would copy +everything and would not give the desired results. + +Indexes +======= + +By default, the index command writes both a full index and an increment index +to a pair of files whose names start with the specified prefix. For example, +the command "index t0" will create files "t0-full.xml" and "t0-incr.xml" in +the user's current directory. (The rationale for writing both indices is that +that one can start from any full index written, apply any number of subsequent +incremental indexes they wish, and the result should be the same as the full +index corresponding to the last incremental index used.) The index command +allow overriding the default behavior by using the "-i" (incremental index +only) or "-f" (full index only) options. To simulate normal system operation, +(and minimize the number of files produced during testing), one could specify +"-f" for the index produced at the start of the run (e.g., "index -f t0"), then +use "-i" for all subsequent indexes up to the last one, and use neither flag +for the final index. This would give a starting and ending full index, and a +full set of incremental indexes to validate against the final full index. + +When testing indexes it's probably a good idea to start a command file with +the command "index [-f] t0" (or the prefix of your choice); this creates a +starting point for incremental indexes from the beginning of the session. +index commands can be interspersed in a command stream at any point, and the +resulting full and incremental indexes can be inspected (and processed +using fsidxupd). + +The fsck command searches the entire file system for any files or directories +with New or Modified flags. This can be used after an index command with the +"-i" (incremental only) flag to verify that all objects with these flags have +been visited, and that all such flags have been correctly reset. + +A pair of "t0" indexes (created at startup) looks like this: + + + 0 + + + + + + + 0 + + + + + +The two are essentially identical, since the root directory is the only one in +existence, and it is new and thus shows up in the incremental index as well. +(Actually, the t0 incremental index is useless, but it helps illustrate how +indexes work.) + +Invoking fssim with this set of commands (in a command file): + + index t0 + mkdir Dir1 + touch Dir2/Dir3/foo + touch Dir2/Dir3/bar + index t1 + touch Dir2/Dir3/baz + rmdir Dir1 + index t2 + +produced the following two "t2" indexes: + + + 0 + + + + Dir2 + 2 + + + + Dir3 + 3 + + + + 5 + bar + + + + 6 + baz + + + + 4 + foo + + + + + + + + + + + + + + Dir1 + + + + Dir2 + + + Dir3 + + + + baz + 6 + + + + + + + + + +Using fsidxupd +============== + +fsidxupd is used to verify the correctness of incremental indexes and their +application to a full index. For example, using the sample list of commands +in the Index section above, three sets of indexes will be created by fssim +when running the commands; the file names will be t0-full.xml, t0-incr.xml, +t1-full.xml, t1-incr.xml, t2-full.xml, and t2-incr.xml. The following +fsidxupd command will take the starting full index and apply the two appro- +priate incremental indexes: + + fsidxupd t0-full.xml t1-incr.xml t2-incr.xml + +The output of this command will be a file named upd-full.xml. In this example, +if the system is working correctly upd-full.xml will be identical to +t2-full.xml. In Unix-like systems, the command: + + diff upd-full.xml t2-full.xml + +would show no output (no differences). + +Program Organization +==================== + +The file fscommon.py contains all of the file system logic. It contains +object definitions for directory and file objects, and implements all of +the low-level logic for managing those objects. (In function, it roughly +corresponds to the vfs module in a Unix-like OS.) It also contains the +the python version-agnostic print function (fsprt). + +The file fsglobals.py defines global variables that must be shared between +the different modules. In order to provide a consistent naming reference +to these shared global variables, they must be in a separate python module +and that module must be imported with the same name in each program. (For +succinctness, I import it as "g", so that global values are referenced as +g.variablename.) + +The file fssim is the command processor for user file system commands; it +processes commands from a command file and/or the command line. (In Unix +terms, it implements the user shell and shell commands.) + +The file fsidxupd is a utility program that reads, updates, and writes +indexes. It uses file system functions from fscommon.py to update a full +index using data from incremental indexes. + +fsruntest +========= + +The programs also include a bash shell script to automate testing. The +format of the command is: + + fsruntest [-d] + +The command file is used to run the test. It may end with an "exit" command, +or the exit command can be entered manually while the test is running. The +"-d" option runs fssim and fsidxupd with debugging messages enabled. + +The script first invokes fssim, passing in the name of the command file. It +then invokes fsidxupd to apply all of the subsequent incremental indexes to +the first full index file created. Finally, it compares the output of fsidxupd +to the final full index created by fssim. If the output of diff is not empty, +the output is preceded by a highlighted error line (so that it stands out or +can be searched for in a large test run). + +Since it is a bash shell script, the test script runs on both Linux and Mac +systems, and with the proper Linux subsystem configured perhaps even on +Windows 10 (or earlier, using Cygwin). + + +Final Comments +============== + +The oids in the files and directories correspond to the fileuid in LTFS. +Initially I did not believe that oids (and thus fileuids) were important +to processing incremental indexes. It turns out that I was wrong, and that +a unique identifier for an object is important to the correct application +of an incremental index in one important case. Timestamps are the last +object modification time. I think all other object fields are obvious. + +The programs have not been intensively debugged. There may be lingering +errors in the commands and/or in index generation. It all seems to work, +but if you find errors (or even think you *may* have found an error) please +let me know. The better debugged the programs are, the more reliable our +results will be. + + -David Pease - pease@coati.com diff --git a/contrib/fssim/src/.gitignore b/contrib/fssim/src/.gitignore new file mode 100644 index 00000000..6722cd96 --- /dev/null +++ b/contrib/fssim/src/.gitignore @@ -0,0 +1 @@ +*.xml diff --git a/contrib/fssim/src/fscommon.py b/contrib/fssim/src/fscommon.py new file mode 100644 index 00000000..4df006c1 --- /dev/null +++ b/contrib/fssim/src/fscommon.py @@ -0,0 +1,713 @@ +# +# File System Simulator for LTFS Common Routines +# +# This module is imported into fssim and fsidxupd, and provides common +# values, objects, and functions used by both programs. +# +# (c) David Pease - IBM Almaden Research Center - May 2018 +# (c) David Pease - pease@coati.com - Sept. 2020 +# + +from copy import copy as pyobjcopy +from datetime import datetime +import fsglobals as g + +useNextOID = None # constant for readability +Log = {} # dictionary to hold update log entries + +########################################################################### +# object flag bit values # +########################################################################### + +modified = 0x01 # object has been modified since last index +new = 0x02 # directory is newly created since last index + +########################################################################### +# Directory object class # +########################################################################### + +class Dir(object): + def __init__(self, parent, oid=None, time=None): + self.oid = oid + self.flags = new + if time: # local code to avoid setting modified attribute + self._modTime = time + else: + self._modTime = str(datetime.now()) + self.parent = parent + self.contents = {} + @property + def oid(self): + return self._oid + @oid.setter + def oid(self, oid): + if oid is not None: + self._oid = oid + else: + self._oid = g.nextoid + g.nextoid += 1 + @property + def parent(self): + return self._parent + @parent.setter + def parent(self, parent): + self._parent = parent + @property + def modTime(self): + return self._modTime + @modTime.setter + def modTime(self, time=None): + if time: + self._modTime = time + else: + self._modTime = str(datetime.now()) + self.isModified = True + @property + def isDir(self): + return True + @property + def isFile(self): + return False + @property + def isModified(self): + return bool(self.flags & modified) + @isModified.setter + def isModified(self, ismodified): + if ismodified: + self.flags |= modified + else: + self.flags &= (0xff - modified) + @property + def isNew(self): + return bool(self.flags & new) + @isNew.setter + def isNew(self, isnew): + if isnew: + self.flags |= new + else: + self.flags &= (0xff - new) + @property + def children(self): + return self.contents.keys() + def contains(self, name): + return name in self.contents + def obj(self, name): + if not name in self.contents: + return None + return self.contents[name] + def addObj(self, name, obj, update=True): + self.contents[name] = obj + if update: + self.modTime = str(datetime.now()) + self.isModified = True + return True + def replObj(self, name, obj): + if name not in self.contents: + return False + self.contents[name] = obj + return True + def rmObj(self, name, verifyEmpty=True, update=True): + if not name in self.contents: + return False + obj = self.contents[name] + if obj.isDir and verifyEmpty: + if len(obj.contents) > 0: + return False + del self.contents[name] + if update: + self.modTime = str(datetime.now()) + self.isModified = True + return True + def __str__(self): + flags = ["new," * self.isNew, "mod," * self.isModified] + str = "Dir: oid %i, flags: '%s', parent:%i, size:%i %s %s" % \ + (self.oid, "".join(flags)[:-1], self.parent.oid, len(self.contents), + self.modTime[5:16], repr(self).split()[-1][2:-1]) + return str + +########################################################################### +# File object class # +########################################################################### + +class File(object): + def __init__(self, parent, oid=None, time=None): + self.oid = oid + self.flags = modified + self.modTime = time + self.parent = parent + self.data = "" + @property + def oid(self): + return self._oid + @oid.setter + def oid(self, oid): + if oid is not None: + self._oid = oid + else: + self._oid = g.nextoid + g.nextoid += 1 + @property + def parent(self): + return self._parent + @parent.setter + def parent(self, parent): + self._parent = parent + @property + def modTime(self): + return self._modTime + @modTime.setter + def modTime(self, time=None): + if time: + self._modTime = time + else: + self._modTime = str(datetime.now()) + self.isModified = True + @property + def isDir(self): + return False + @property + def isFile(self): + return True + @property + def isModified(self): + return bool(self.flags & modified) + @isModified.setter + def isModified(self, ismodified): + if ismodified: + self.flags |= modified + else: + self.flags &= (0xff - modified) + @property + def data(self): + return self._data + @data.setter + def data(self, data): + self._data = data if data else "" + def __str__(self): + flag = "mod" * self.isModified + str = "File: oid %i, flags:'%s', parent:%i, size:%i %s %s" % \ + (self.oid, flag, self.parent.oid, len(self.data), + self.modTime[5:16], repr(self).split()[-1][2:-1]) + return str + +########################################################################### +# python version-agnostic print function # +########################################################################### + +def fsprt(*args): + outstr = "" + for arg in args: + outstr += (str(arg) + " ") + print(outstr) + +########################################################################### +# debugging-only print function # +########################################################################### + +def dbprt(*args): + if g.debug: + fsprt(*args) + +########################################################################### +# create a log entry # +########################################################################### + +def log(name, oid, reason): + logkey = name + "#" + str(oid) + # don't overwrite a "New" log entry with a "Mod" one for the same object + if logkey in Log and Log[logkey][0] == "New": + if reason == "Mod": + dbprt("Skip log: Mod for", name, "- is already New") + return + # trim unneeded log entries if parents are New or Deleted + # (processing log entries in reverse time order - why does it matter??) + for entkey in sorted(Log, key=lambda entkey: Log[entkey][1], reverse=True): + entpath, entoid = entkey.rsplit("#", 1) + entreason, enttime = Log[entkey] + # if a parent dir is already "New", don't bother to log updates + if entreason == "New" and name.startswith(joinPath(entpath,"/")): + dbprt("Skip log", reason, "for", name, "- parent is New") + return + # if this is a parent being deleted, remove unneeded entries for children + elif reason == "DelD": + if entpath.startswith(joinPath(name, "/")): + dbprt("Remove log", entreason, "for", entpath, "- parent Deleted") + del Log[entkey] + Log[logkey] = (reason, datetime.now()) + +########################################################################### +# display log entries in pathname (processing) or time order # +########################################################################### + +def printLog(timeOrder=False): + if timeOrder: + sortkey = lambda k: Log[k][1] + else: + sortkey = None + for k in sorted(Log, key=sortkey): + if timeOrder: + fsprt(str(Log[k][1])[:-7], Log[k][0], k) + else: + fsprt("%-20s" % k, Log[k][0], str(Log[k][1])[:-7]) + return True + +########################################################################### +# file system common utility functions # +########################################################################### + +# return a fully-qualified name (with no trailing slash) +def fullName(name): + dbprt("fullName in:", name) + if name.startswith("/"): + fullname = name + else: + fullname = joinPath(g.curnm, name) + if len(fullname) > 1 and fullname.endswith("/"): + fullname = fullname[:-1] + dbprt("fullName out:", fullname) + return fullname + +# return full path and name portions of a filespec +def splitPath(name): + path,name = fullName(name).rsplit("/",1) + if not path: + path = "/" + return path,name + +# return a properly joined path and file name +def joinPath(path, name): + if name.startswith("/"): + name = name[1:] + if path.endswith("/"): + return path + name + else: + return path + "/" + name + +# process a directory path, output depends on parentRef value: +# False returns: full name of object, object instance (default behavior) +# True returns: full path to object, parent object instance, object name +def dirpath(name, parentRef=False): + fullname = fullName(name) + if not parentRef and fullname != "/": + fullname += "/" + dbprt("dirpath in:", fullname) + wdir = g.root + wdnm = "/" + parts = fullname[1:].split("/") + if parts[-1]: + lastpart = len(parts)-1 + else: + lastpart = len(parts)-2 + for cnt, dn in enumerate(parts[:-1]): + if wdir.contains(dn): + if wdir.obj(dn).isDir or cnt == lastpart: + wdir = wdir.obj(dn) + wdnm = joinPath(wdnm, dn) + continue + dbprt("dirpath out: None") + if parentRef: + return None, None, None + else: + return None, None + dbprt("dirpath out:", wdnm, wdir.oid, parts[-1]) + if parentRef: + return wdnm, wdir, parts[-1] + else: + return wdnm, wdir + +# depth-first-search of a file system tree with function invocation on descent +# parameters: +# sdobj - starting directory object (top of dfs search tree) +# sdnm - the fully-qualified name of the starting directory +# func - the function to be invoked while traversing the tree +# params - a tuple of parameters to pass to func +# return value: +# boolean - from "enter" into a directory, flag indicating whether to +# descend into (that is process the members of) that directory, +# otherwise flag from func that halts processing of dir if False +# parameters passed to func: +# reason - one of "enter", "exit", "file", or "dir" +# sdobj, sdnm, object name if "file" or "dir" else None, params +def dfs(sdobj, sdnm, func, params): + if sdnm == "/": + name = "/" + else: + name = splitPath(sdnm)[1] + descend = func("enter", sdobj, sdnm, None, params) + if descend: + for name in sorted(sdobj.children, # process files before subdirs + key=lambda name: 'x\ff'+name if sdobj.obj(name).isDir else name): + if sdobj.obj(name).isFile: + if not func("file", sdobj, sdnm, name, params): + return False + else: + if not func("dir", sdobj, sdnm, name, params): + return False + wdir = sdobj.obj(name) + wdnm = joinPath(sdnm, name) + if not dfs(wdir, wdnm, func, params): + return False + return func("leave", sdobj, sdnm, None, params) + else: + return True + +# create a new directory or file obj, add it to parent, return object instance +def makeObj(name, dir=False, oid=None, ts=None, update=True): + path, parent, newname = dirpath(name, parentRef=True) + if parent == None: + return None + if parent.contains(newname): + return None + if dir: + newobj = Dir(parent, oid=oid, time=ts) + state = "New" + else: + newobj = File(parent, oid=oid, time=ts) + state = "Mod" + log(joinPath(path, newname), newobj.oid, state) + parent.addObj(newname, newobj, update=update) + return newobj + +# remove a directory or file +def rmObj(name, dir=False, update=True): + if name == "/": + return False # can't remove root + path, parent, oldname = dirpath(name, parentRef=True) + if parent == None or not parent.contains(oldname): + return False + obj = parent.obj(oldname) + if dir: + if obj.isFile: + return False + if obj == g.curdir: + return False # can't remove current dir + else: + if obj.isDir: + return False + if not parent.rmObj(oldname, update=update): # for dir, parent will if empty + return False + deltype = "DelD" if dir else "DelF" + log(joinPath(path, oldname), obj.oid, deltype) + return True + +# print info about an object, including memory address (for debugging only) +def printObj(obj, prefix="", children=False): + fsprt(prefix+str(obj)) + if children and obj.isDir: + for child in obj.children: + fsprt(" "+str(obj.obj(child))) + +# move or rename a directory or file, or copy a single file +def movecopy(srcname, tgtname, copy=False): + # get full paths of source and target + if srcname == "/": # can't move root, copying it would be recursive + return False + if tgtname == ".": + tgtname = g.curnm + fulltgt = tgtname # save original name for later + # get source and target object and name information + srcpath, srcobj = dirpath(srcname) + tgtpath, tgtparent, tgtname = dirpath(tgtname, parentRef=True) + if not srcobj or not tgtparent: + return False + # don't allow recursive copy + if tgtpath.startswith(srcpath+"/"): + return False + # cannot copy a complete directory + if copy and srcobj.isDir: + return False + dbprt("source:", srcpath, srcobj.oid) + dbprt("target:", tgtpath, tgtparent.oid, tgtname) + srcname = splitPath(srcpath)[1] + # if target is simply "/", must have a file name to continue + if not tgtname: + tgtname = srcname + # if target is a directory, make it new parent and copy source filename + if tgtparent.contains(tgtname) and tgtparent.obj(tgtname).isDir: + tgtpath, tgtparent = dirpath(fulltgt) + tgtname = srcname + dbprt("new target info:", tgtpath, tgtname, tgtparent.oid) + # now that we have all the object info, make sure not moving object to self + srcparent = srcobj.parent + if srcparent == tgtparent and srcname == tgtname: + dbprt("Move to self!") + return False + # if target already exists, decide what needs to be done (quite complex!) + remtobj = None # flag to delete old target object + if tgtparent.contains(tgtname): + # if the target is a directory + if tgtparent.obj(tgtname).isDir: + # and the source is a file + if srcobj.isFile: + # if target contains new object name, it must be a file + if tgtparent.obj(tgtname).contains(srcname): + if tgtparent.obj(tgtname).obj(srcname).isDir: + return False + # it's a file, so we must remove it + else: + remtobj = tgtparent.obj(tgtname).obj(srcname) + # the target is a file + else: + # if the source is a directory, can't move it to a file + if srcobj.isDir: + return False + # source is a file + if not copy: + remtobj = tgtparent.obj(tgtname) + dbprt("tgt exists:", ("remove "+str(remtobj.oid)) if remtobj else "") + # we're finally ready to move something + dbprt("mvcp:", srcpath,srcname,srcobj.oid, tgtpath,tgtname,tgtparent.oid) + # if copy, create a copy of the source object + if copy: + newobj = pyobjcopy(srcobj) # new python object, copy of srcobj + newobj.oid = useNextOID + else: # move (move original object, make copy in source for deletion) + newobj = srcobj # newobj is srcobj (same python object) + srcobj = pyobjcopy(newobj) # create new python object copy for srcobj + if not srcparent.replObj(srcname, srcobj): # replace srcobj ref in parent + return False + # set target object's parent and state + newobj.parent = tgtparent + if newobj.isDir: + newobj.isNew = True + status = "New" + else: + newobj.isModified = True + status = "Mod" + # if necessary, remove existing file that is being replaced + if remtobj: + remoid = tgtparent.obj(tgtname).oid + if not tgtparent.rmObj(tgtname): + return False + log(joinPath(tgtpath, tgtname), remoid, "DelF") + # put moved/copied object in its new home + tgtparent.addObj(tgtname, newobj) # put object under target name + log(joinPath(tgtpath, tgtname), newobj.oid, status) + # for move, remove object from its old home + if not copy: + if not srcparent.rmObj(srcname, verifyEmpty=False): # rem src from parent + fsprt("Error: cannot remove source object for move; object copied.") + return False + deltype = "DelF" if srcobj.isFile else "DelD" + dbprt("Creating log:", deltype, srcpath, srcobj.oid) + log(srcpath, srcobj.oid, deltype) + if g.debug: + printObj(newobj, "mvcp new: ") + printObj(srcobj, "mvcp src: ") + return True + +# copy a directory's contents to an existing directory recursively +def copyrecurs(inloc, outloc): + # get source and target object and name information + if outloc == ".": + outloc = g.curnm + srcpath, srcobj = dirpath(inloc) + tgtpath, tgtobj = dirpath(outloc) + if not srcobj or not tgtobj: + return False + # don't allow recursive copy + if tgtpath.startswith(srcpath+"/"): + return False + # both source and target must be existing directories + if not srcobj.isDir or not tgtobj.isDir: + return False + # recursively copy files and directories + return dfs(srcobj, srcpath, copyObj, (srcpath, tgtpath)) + +# make a copy of a single file or directory (invoked through dfs()) +def copyObj(desc, dirobj, path, name, moreparams): + srcfname, tgtfname = moreparams + if desc not in ("file", "dir"): + return True + dbprt("copyObj:", desc, dirobj.oid, path, name, srcfname, tgtfname) + addpath = path[len(srcfname):] + if addpath: + tgtdir = tgtfname + addpath + else: + tgtdir = tgtfname + newname = joinPath(tgtdir, name) + if desc == "file": + oldname = joinPath(path, name) + dbprt("Copying file", oldname, "to", newname) + return movecopy(oldname, newname, copy=True) + elif desc == "dir": + dbprt("Creating directory", newname) + if not makeObj(newname, dir=True): + return False + # should copy other attributes from source to new dir here, if needed + return True + +########################################################################### +# routines used in generating indexes # +########################################################################### + +# write a full index to a file (uses dfs to descend entire tree) +def fullIndex(fn): + global indexLevel, Log + fullfn = fn + "-full.xml" + with open(fullfn, "w") as fh: + fh.write("\n") + indexLevel = 0 # used only for indentation of xml + indexEnt("dir", g.root, "/", None, ("full", fh)) + result = dfs(g.root, "/", indexEnt, ("full", fh)) + fh.write("\n") + Log = {} # clear log after writing an index + if not result: + fsprt("Error occurred creating full index") + return result + +# write an incremental index to a file +def incrIndex(fn): + global indexLevel, Log + success = True + incrfn = fn + "-incr.xml" + with open(incrfn, "w") as fh: + fh.write("\n") + indexLevel = 0 # used for indentation and to close paths + # always start with an entry for the root dir, whether or not it's in log + indexEnt("dir", g.root, "/", "", ("incr", fh)) + indexLevel += 1 + lastpath = "/" + pathindir = None + # process all log entries sorted in name,oid order + for logKey in sorted(Log): + # get full name, oid, log entry type and timestamp for this log entry + fullname, oid = logKey.rsplit("#", 1) + if fullname == "/" and oid == "0": # we already did the root dir + continue + path, name = splitPath(fullname) + reason, ts = Log[logKey] + # if debugging, show log entry + dbprt("Log entry: %s (%i) %s %s\n" % (fullname, int(oid), reason, ts)) + # get object reference for log target + obj = dirpath(fullname)[1] # already have full name, need obj + # didn't find object for this entry, error unless it was for deletion + if not obj and not reason.startswith("Del"): + dbprt("Object matching log entry not found", fullname, oid) + success = False + break + # found the same name but a different oid, also bad unless delete + if not reason.startswith("Del") and obj.oid != int(oid): + dbprt("Object oids do not match:", fullname, oid, obj.oid) + continue + # close out unneeded directories from prior object + lastdirs = lastpath.split("/")[1:] if lastpath != "/" else [] + newdirs = path.split("/")[1:] if path != "/" else [] + newLevel = len(newdirs) + lastmatch = -1 + for i in range(min(len(lastdirs), len(newdirs))+1): + if len(lastdirs) > i and len(newdirs) > i: + if lastdirs[i] == newdirs[i]: + continue + lastmatch = i-1 + break + i = 0 # inialize in case following loops null + for i in range(len(lastdirs)-1, lastmatch, -1): + fh.write(" " * (i+1) + " \n") + fh.write(" " * (i+1) + "\n") + # build new directory path structure as needed for object + for i in range(lastmatch+1, newLevel): + prefix = " " * (i+1) + temppath = "/".join(['']+newdirs[:i+1]) # full path name of dir + if temppath == pathindir: # if we just added this dir + continue # to the index, skip it + fh.write(prefix + "\n") + fh.write(prefix + " " + newdirs[i] + "\n") + tempobj = dirpath(temppath)[1] + if tempobj.isModified: # parent dir may have updated modtime + fh.write(prefix + " \n") + tempobj.isModified = False + fh.write(prefix + " \n") + indexLevel = newLevel + 1 + pathindir = None + # process Deletes here rather than using indexEnt() + if reason.startswith("Del"): + lastpath = path # make sure to save last path for deletes as well + if obj: + if obj.oid == int(oid): + dbprt("Found deleted object in file system", fullname, oid) + success = False + break + else: + continue # don't delete object if it's been recreated + prefix = " " * indexLevel + indextype = "file" if reason == "DelF" else "directory" + fh.write(prefix + "<" + indextype + ">\n") + fh.write(prefix + " " + name + "\n") + fh.write(prefix + " \n") + fh.write(prefix + "\n") + continue + # write the object entry into the index + newdir = obj.isDir and obj.isNew # indexEnt() will reset this + indextype = "file" if obj.isFile else "dir" + indexdirobj = obj.parent if obj.parent else g.root + indexEnt(indextype, indexdirobj, fullname, name, ("incr", fh)) + # if object was a New directory, output its full subtree + if obj.isDir: + if newdir: + if not dfs(obj, fullname, indexEnt, ("full", fh)): + dbprt("Error occurred in dfs for incremental index") + success = False + break + else: + pathindir = joinPath(path, name) # this feels like a kludge! + # save index state from this entry to compare to the next + lastpath = path + # close out the directory tree and the incremental index + for i in range(indexLevel-1, 0, -1): + fh.write(" " * i + " \n") + fh.write(" " * i + "\n") + fh.write(" \n") + fh.write("\n") + # clear the log and return + Log = {} + if not success: + # delete incremental index file, but for now keep it for debugging + fsprt("Error occurred creating incremental index") + return success + +# write the XML for single index entry (this function my be invoked directly, +# but is often invoked through dfs(), thus the use of params) +def indexEnt(desc, dirobj, path, name, params): + global indexLevel + enttype, fh = params + dbprt("Index entry:", desc, dirobj.oid, path, name, enttype) + if not name: # entering or leaving a directory, or root dir + obj = dirobj # obj is the directory itself + if path != "/": + name = splitPath(path)[1] + else: + obj = dirobj.obj(name) # obj is a member of the directory + if desc == "enter": + indexLevel += 1 + elif desc == "file": + prefix = " " * indexLevel + fh.write(prefix + "\n") + fh.write(prefix + " " + name + "\n") + fh.write(prefix + " " + str(obj.oid) + "\n") + # note: if enttype == "incr", should only output changed attributes + fh.write(prefix + " \n") + if obj.data or enttype == "incr": # if incr, replace any old data + fh.write(prefix + " " + obj.data + "\n") + fh.write(prefix + "\n") + obj.isModified = False + elif desc == "dir": + prefix = " " * indexLevel + if obj != g.root: + fh.write(prefix + "\n") + fh.write(prefix + " " + name + "\n") + if enttype == "full" or obj.isNew: # ??? does this match spec? + fh.write(prefix + " " + str(obj.oid) + "\n") + fh.write(prefix + " \n") + fh.write(prefix + " \n") + obj.isModified = False + obj.isNew = False + elif desc == "leave": + indexLevel -= 1 + prefix = " " * indexLevel + if enttype == "full" or obj != g.root: + fh.write(prefix + " \n") + if obj != g.root: + fh.write(prefix + "\n") + return True + diff --git a/contrib/fssim/src/fsglobals.py b/contrib/fssim/src/fsglobals.py new file mode 100644 index 00000000..12d1d84b --- /dev/null +++ b/contrib/fssim/src/fsglobals.py @@ -0,0 +1,10 @@ +# +# fssim, fsidxupd, fscommon shared global variables +# + +debug = False # debugging mode flag +nextoid = 0 # master object id +root = None # root directory object +curdir = None # current directory object +curnm = None # current directory path string + diff --git a/contrib/fssim/src/fsidxupd b/contrib/fssim/src/fsidxupd new file mode 100755 index 00000000..06991f42 --- /dev/null +++ b/contrib/fssim/src/fsidxupd @@ -0,0 +1,159 @@ +#!/usr/bin/python3 +# +# File System Index Processor for LTFS +# +# This program reads in a full index created by the fssim program, then +# updates that index using one or more incremental indexes created +# sequentially by the same program. At the end, it writes the resulting +# xml index to a file (upd-full.xml). +# +# Usage: +# +# fsidxupd [-d] [ ...] +# +# where: +# -d: prints lots of debugging messages during processing +# fullindex: a file containing a full index (from fssim) +# incrindex: a file containing an incremental index (from fssim) +# +# (c) David Pease - IBM Almaden Research Center - May 2018 +# (c) David Pease - pease@coati.com - Sept. 2020 +# + +from sys import argv, exit +from xml.etree.ElementTree import parse +from fscommon import * +import fsglobals as g + +########################################################################### +# process xml children (files and dirs) in blocks recursivley # +########################################################################### + +def processContents(path, contents, full=False): + + for child in contents: + values = {} + for val in child: + if val.tag == "contents": + values[val.tag] = val + else: + values[val.tag] = val.text + name = joinPath(path, values["name"]) + dir = child.tag == "directory" + if full: + obj = makeObj(name, oid=values["oid"], ts=values["time"], dir=dir, + update=False) + if not obj: + return False + if "data" in values: + obj.data = values["data"] + else: # is incremental + fullpath, obj = dirpath(name) + if not obj: + if "deleted" in values: + pass # already deleted, so okay + else: + dbprt("Creating:", name, values) + if not "oid" in values: + fsprt("Error creating %s, no oid in index" % name) + return False + obj = makeObj(name, oid=values["oid"], ts=values["time"], + dir=dir, update=False) + if not obj: + return False + if "data" in values: + obj.data = values["data"] + else: + if "deleted" in values: + obj.parent.rmObj(values["name"], update=False,verifyEmpty=False) + else: + if "oid" in values and obj.oid != values["oid"]: + if not obj.parent.rmObj(values["name"], update=False, + verifyEmpty=False): + return False + obj = makeObj(name, oid=values["oid"], ts=values["time"], + dir=dir, update=False) + if not obj: + return False + else: + if "time" in values: + obj.modTime = values["time"] + if "data" in values: + obj.data = values["data"] + if dir and not "deleted" in values: + dbprt("Entering:", child.tag, values) + if not processContents(name, values["contents"], full=full): + return False + return True + +########################################################################### +# main line # +########################################################################### + +# do some parameter validation +if len(argv) > 1 and argv[1] == "-d": + g.debug = True + del argv[1] +if len(argv) < 2: + fsprt("Too few parameters") + exit(12) +fullfn = argv[1] +incrlist = argv[2:] + +# first build the base filesystem from the full index +try: + xml = parse(fullfn) +except: + fsprt("Error parsing XML in file", fullfn) + exit(12) +xmlroot = xml.getroot() +if xmlroot.tag != "fullindex": + fsprt("Invalid full index file header", xmlroot.tag) + exit(12) +oid = ts = contents = None +for child in xmlroot: + if child.tag == "oid": + oid = int(child.text) + elif child.tag == "time": + ts = child.text + elif child.tag == "contents": + contents = child + else: + fsprt("Invalid tag", tag) +if oid == None or not ts or contents is None: + fsprt("Missing root directory entries") + exit(12) +g.root = Dir(None, oid=oid, time=ts) +g.curdir = g.root +g.curnm = "/" +if not processContents("/", contents, full=True): + fsprt("Error creating full index structure.") + exit(12) + +# now start applying updates from incremental indexes +for incrfn in incrlist: + try: + xml = parse(incrfn) + except: + fsprt("Error parsing XML in file", incrfn) + exit(12) + xmlroot = xml.getroot() + if xmlroot.tag != "incrementalindex": + fsprt("Invalid incremental index file header", xmlroot.tag) + exit(12) + ts = contents = None + for child in xmlroot: + if child.tag == "time": + ts = child.text + elif child.tag == "contents": + contents = child + if ts: + g.root.modTime = ts + if contents is not None: + if not processContents("/", contents): + fsprt("Error applying incremental index", incrfn) + exit(12) + +# write out final results as a full index +fullIndex("upd") + diff --git a/contrib/fssim/src/fsruntest b/contrib/fssim/src/fsruntest new file mode 100755 index 00000000..4f9e0e79 --- /dev/null +++ b/contrib/fssim/src/fsruntest @@ -0,0 +1,74 @@ +#!/bin/bash +# +# Run an fssim test case, apply incremental indexes to initial full index, +# compare result to final full index, and display any difference +# + +# process operands and set values +if [ $# -lt 1 ]; then + echo "Usage:" $0 "[-d] " + exit 1 +fi +if [ $1 == "-d" ]; then + OPT="-d" + shift 1 +else + OPT="" +fi +if [ $# -ne 1 ]; then + echo "Missing command file name" + exit 1 +fi +if [ ! -f $1 ]; then + echo "File" $1 "does not exist" + exit 1 +fi + +# initialize xml file name suffixes +FNAME="-full.xml" +INAME="-incr.xml" + +# create list of indexes written by input command file +FLIST="" +while read -r prefix; do + FLIST="$FLIST $prefix" +done < <(grep "^ *index" $1 | rev | cut -d" " -f 1 | rev) + +# verify that at least two indexes are written by test case +alist=( $FLIST) +if [ ${#alist[@]} -lt 2 ]; then + echo "Too few indexes written in test script $1 for comparison" + exit 1 +fi + +# build file name strings based on list of indexes and name suffixes +FULLFILE="" +INCRFILES="" +for prefix in $FLIST; do + if [[ $FULLFILE == "" ]]; then + FULLFILE=$prefix$FNAME + else + INCRFILES="$INCRFILES $prefix$INAME" + COMPFILE=$prefix$FNAME + fi +done + +# build the actual command lines +CMD1="./fssim $OPT $1" +CMD2="./fsidxupd $OPT $FULLFILE $INCRFILES" +CMD3="diff upd-full.xml $COMPFILE" + +# display each command, then execute it +echo $CMD1 +$CMD1 +echo $CMD2 +$CMD2 +echo $CMD3 +$CMD3 > tempout.txt + +# check for errors and display them +if [ -s tempout.txt ]; then + echo "************** Errors from diff - $1 *****************" + cat tempout.txt +fi +rm tempout.txt diff --git a/contrib/fssim/src/fssim b/contrib/fssim/src/fssim new file mode 100755 index 00000000..be368613 --- /dev/null +++ b/contrib/fssim/src/fssim @@ -0,0 +1,376 @@ +#!/usr/bin/python3 +# +# File System Simulator for LTFS +# +# This program simulates file system directory activity such as creating or +# deleteing files and directories, moving and copying files or directories, +# listing information, etc. It also writes full and incremental XML indexes +# to files on request. +# +# Usage: +# +# fssim [-d] [command file] +# +# where: +# -d: prints lots of debugging messages during processing +# command file: executes commands from specified text file at startup +# +# (c) David Pease - IBM Almaden Research Center - May 2018 +# (c) David Pease - pease@coati.com - Sept. 2020 +# + +from sys import argv, exit, version +from fscommon import * +import fsglobals as g +import readline + +TimeNow = None # a constant value for readability + +########################################################################### +# filesystem command functions # +########################################################################### + +# cd: change directory +def cd(opt, name=None): + if name == "/" or name == None: + g.curdir = g.root + g.curnm = "/" + return True + if name == ".." and g.curnm != "/": + g.curdir = g.curdir.parent + g.curnm = splitPath(g.curnm)[0] + if g.curnm == "": + g.curnm = "/" + return True + dirnm, newdir = dirpath(name) + if newdir == None: + return False + g.curdir = newdir + g.curnm = dirnm + return True + +# md: create a directory or (multi-level) directory path +def md(opt, *names): + for name in names: + fullname = fullName(name) + pieces = fullname[1:].split("/") + path = "" + for piece in pieces: + path += ("/" + piece) + fpath, obj = dirpath(path) + if not obj: + if not makeObj(path, dir=True): + return False + return True + +# rd: remove a directory +def rd(opt, name): + return rmObj(name, dir=True) + +# rm: remove a file +def rm(opt, name): + return rmObj(name) + +# cp: copy a single file, or copy a directory's contents recursively +def cp(opt, oldname, newname): + if opt: + if opt == "-r": + return copyrecurs(oldname, newname) + else: + return False + return movecopy(oldname, newname, copy=True) + +# mv: move a file or directory +def mv(opt, oldname, newname): + return movecopy(oldname, newname) + +# tf: update an object's timestamp or create a new file +def tf(opt, *names): + for name in names: + if name == "/": # special case for touching root dir + g.root.modTime = TimeNow + log(name, 0, "Mod") + continue + path, parent, objname = dirpath(name, parentRef=True) + if parent == None: + if not md(None, splitPath(name)[0]): + return False + path, parent, objname = dirpath(name, parentRef=True) + if parent == None: + return False + if not parent.contains(objname): + if not makeObj(name): + return False + else: + parent.obj(objname).modTime = TimeNow + log(joinPath(path, objname), parent.obj(objname).oid, "Mod") + return True + +# ec: echo string to a file or the screen +def ec(opt, *args): + if len(args) >= 2 and args[-2] in (">", ">>"): + name = args[-1] + path, obj = dirpath(name) + if obj: + if obj.isDir: + return False + obj.modTime = TimeNow + else: # need to create file first + if not tf(None, name): + return False + path, obj = dirpath(name) + if not obj: + return False + data = " ".join(args[:len(args)-2]) + if args[-2] == ">>": + obj.data += data + else: + obj.data = data + log(path, obj.oid, "Mod") + else: + fsprt(*args) + return True + +# ct: cat (print) a file's data +def ct(opt, name): + path, obj = dirpath(name) + if not obj or obj.isDir: + return False + fsprt(obj.data) + return True + +# ls: list directory contents or file information with optional information +def ls(opt, dirnm=None, showall=False): + if opt: + if opt == "-a": + showall = True + else: + return False + if not dirnm: + wobj = g.curdir + dirnm = g.curnm + else: + path, wobj = dirpath(dirnm) + if not wobj: + return False + if wobj.isFile: + flag = "-m-" if wobj.isModified else "---" + lth = len(wobj.data) + fsprt(flag, wobj.oid, wobj.modTime[5:16], lth, splitPath(path)[1]) + return True + if showall: + flag = "d" +"-m"[wobj.isModified] +"-n"[wobj.isNew] + fsprt(flag, wobj.oid, wobj.modTime[5:16], len(wobj.children), ".") + for name in sorted(wobj.children, # process subdirs before files + key=lambda name: "\0"+name if wobj.obj(name).isDir else name): + if wobj.obj(name).isDir: + flag = "d" + "-m"[wobj.obj(name).isModified] + \ + "-n"[wobj.obj(name).isNew] + lth = len(wobj.obj(name).children) + else: + flag = "-" + "-m"[wobj.obj(name).isModified] + "-" + lth = len(wobj.obj(name).data) + fsprt(flag, wobj.obj(name).oid, wobj.obj(name).modTime[5:16], lth, name) + return True + +# ll: list long, same as ls -a +def ll(opt, dirnm=None): + return ls(opt, dirnm, showall=True) + +# tr: show a tree view of file system with optional oids +def tr(opt): + oids = False + if opt: + if opt == "-o": + oids = True + else: + return False + if not oids: + fsprt("/") + else: + fsprt("/ (0)") + return dfs(g.root, "/", tree, oids) + +# print out one line of the tree view (invoked through dfs()) +def tree(desc, dirobj, path, name, oids): + if desc not in ("file", "dir"): + return True + if path == "/": + prefix = "" + else: + prefix = path.count("/") * " " + if oids: + oidstr = " (" + str(dirobj.obj(name).oid) + ")" + else: + oidstr = "" + if desc == "file": + fsprt(prefix + " |-" + name + oidstr) + elif desc == "dir": + fsprt(prefix + " |=" + name + oidstr) + return True + +# cl: clear the screen +def cl(opt): + fsprt("\x1b\x5b\x48\x1b\x5b\x32\x4a") + return True + +# ix: write out both a full and an incremental index +def ix(opt, fnprefix): + full = True + incr = True + if opt: + if opt == "-f": + incr = False + elif opt == "-i": + full = False + else: + return False + if incr: + result = incrIndex(fnprefix) + if not result: + full = True # if incremental fails, force full index + if full: # full index clears log, all file flags, so must be last + result = fullIndex(fnprefix) + return result + +# pl: display update log entries by name or optionally by time +def pl(opt): + byTime = False + if opt: + if opt == "-t": + byTime = True + else: + return False + return printLog(byTime) + +# fc: check file system for extraneous or incorrect information +def fc(opt): + if g.root.isNew or g.root.isModified: + fsprt("/", ("Mod" * g.root.isModified), ("New" * g.root.isNew)) + return dfs(g.root, "/", chkMods, None) + +# display objects that indicate New or Modified (invoked thru dfs()) +def chkMods(desc, dirobj, path, name, dummy): + if desc not in ("file", "dir"): + return True + obj = dirobj.obj(name) + mod = obj.isModified + new = obj.isDir and obj.isNew + if mod or new: + fsprt(joinPath(path, name), ("Mod" * mod), ("New" * new)) + return True + +########################################################################### +# execute a single input command (from command file or keyboard) # +########################################################################### + +def execCommand(input): + if input == "exit": + return False + if input == "help": + fsprt("Enter 'exit', 'help', or one of the following commands:") + for cmdname in sorted(cmdTable): + fsprt("%-5s %s" % (cmdname, cmdTable[cmdname][3])) + return True + # tokenize input string allowing quoted strings + sstr = input.split('"') + if len(sstr) % 2 != 1: + fsprt("** Invalid") + return True + input = [] + for ct, ss in enumerate(sstr): + if not ss: + continue + if ct % 2 == 1: + input.append(ss) + continue + else: + input += ss.split() + # process the command + cmdnm = input[0] + if cmdnm not in cmdTable: + fsprt("** Invalid") + return True + func = cmdTable[cmdnm][2] + opt = None + if len(input) > 1 and input[1].startswith("-"): + opt = input[1] + del input[1] + if (len(input)-1) < cmdTable[cmdnm][0] or \ + (len(input)-1) > cmdTable[cmdnm][1]: + fsprt("** Invalid") + return True + params = [] + for p in input[1:]: + params.append(p) + if not func(opt, *params): + fsprt("** Failed") + return True + +########################################################################### +# command table - name: (min operands, max operands, function, help text) # +########################################################################### + +cmdTable = { + # cmd min max fn help text + "cd": (0, 1, cd, "[dirname] - change current directory"), + "mkdir": (1,20, md, "dirname ... - make directories"), + "md": (1,20, md, "dirname ... - alias for mkdir"), + "touch": (1,20, tf, "name ... - create file(s) or update time(s)"), + "rmdir": (1, 1, rd, "dirname - remove directory"), + "rd": (1, 1, rd, "dirname - alias for rmdir"), + "rm": (1, 1, rm, "filename - remove file"), + "cp": (2, 2, cp, "[-r] name name - copy one file, or dir recursively"), + "mv": (2, 2, mv, "name name - move/rename file or directory"), + "ls": (0, 1, ls, "[-a] [name] - list file or directory info"), + "dir": (0, 1, ls, "[-a] [name] - alias for ls"), + "ll": (0, 1, ll, "[name] - alias for ls -a (ls with all info)"), + "tree": (0, 0, tr, "[-o] - show a tree view of the file system"), + "echo": (1, 20,ec, "data [>|>> name] - write data to screen or file"), + "cat": (1, 1, ct, "name - write a file's contents to the screen"), + "cls": (0, 0, cl, " - clear the screen"), + "index": (1, 1, ix, "[-f|-i] fileprefix - write full and/or incr indexes"), + "log": (0, 0, pl, "log [-t] - display log entries by name or time"), + "fsck": (0, 0, fc, " - check files and dirs for correctness"), +} + +########################################################################### +# main line # +########################################################################### + +# create root directory with no parent and make it the current dir +g.root = Dir(None) # equivalent to mkfs -t fssim ;-) +log("/", 0, "New") +g.curdir = g.root +g.curnm = "/" + +# set debugging flag +g.debug = "-d" in argv[1:] + +# if command file specified, execute commands from it +if len(argv) > 1 and not argv[-1].startswith("-"): + infname = argv[-1] + with open(infname, "r") as infile: + inlines = infile.readlines() + for inpt in inlines: + inpt = inpt.strip() + if inpt: + if g.debug: + fsprt("==============================") + if inpt.startswith("#"): + fsprt(inpt) + else: + fsprt(g.curnm+"> "+inpt) + if not execCommand(inpt): + exit(0) + +# go into interactive mode +fsprt('Enter command, "exit", or "help":') +while True: + if version.startswith("2"): + input = raw_input # use python 3.x function name + inpt = input(g.curnm+"> ").strip() + if inpt and not inpt.startswith("#"): + if not execCommand(inpt): + exit(0) + diff --git a/contrib/fssim/testcases/daptest.txt b/contrib/fssim/testcases/daptest.txt new file mode 100644 index 00000000..75dfaee5 --- /dev/null +++ b/contrib/fssim/testcases/daptest.txt @@ -0,0 +1,11 @@ +index t0 +touch a/b/c/foo +log +index t1 +touch a/b/bar +touch a/b/baz +ll a/b +log +index t2 +exit + diff --git a/contrib/fssim/testcases/daptest2.txt b/contrib/fssim/testcases/daptest2.txt new file mode 100644 index 00000000..7f97175f --- /dev/null +++ b/contrib/fssim/testcases/daptest2.txt @@ -0,0 +1,21 @@ +index t0 +ls +ls -a +touch a/b/c/foo +touch a/b/c/bar +log +ls -a +ls -a b +ls -a a/b +md z +ls -a +ls -a a/b/c +index t1 +ll +ll a +md c +ll +ls +rd c +index t2 +exit diff --git a/contrib/fssim/testcases/daptest3.txt b/contrib/fssim/testcases/daptest3.txt new file mode 100644 index 00000000..58144bd5 --- /dev/null +++ b/contrib/fssim/testcases/daptest3.txt @@ -0,0 +1,10 @@ +index t0 +touch foo +echo "Hi" > bar +ll +log +mv bar foo +ll +log +index t1 +exit diff --git a/contrib/fssim/testcases/daptest4.txt b/contrib/fssim/testcases/daptest4.txt new file mode 100644 index 00000000..22cfad72 --- /dev/null +++ b/contrib/fssim/testcases/daptest4.txt @@ -0,0 +1,7 @@ +index t0 +touch a/b/foo +touch a/c/foo +index t1 +touch a/c/foo +index t2 +exit diff --git a/contrib/fssim/testcases/daptest5.txt b/contrib/fssim/testcases/daptest5.txt new file mode 100644 index 00000000..130a219f --- /dev/null +++ b/contrib/fssim/testcases/daptest5.txt @@ -0,0 +1,34 @@ +index t30 +# /A +# |_ /B +# | |_ n1 +# |_ n2 +# |_ n3 +# /C +# |_ n4 +# |_ n5 +# /n1 +# /D +# /E +# +touch /A/B/n1 /A/n2 /A/n3 /C/n4 /C/n5 n1 +mkdir D E +tree +index t31 +# 1. Delete /A/B/n1 +# Rename the directory /C to /A/B/n1 +rm /A/B/n1 +mv /C /A/B/n1 +tree -o +index t32 +# 2. Append data data to /A/B/n1 +# Delete /A/B/n1 +# Rename file /n1 to /A/B/n1 +touch /A/B/n1 +rm /A/B/n1 +mv n1 /A/B/n1 +tree -o +log +index t33 +exit + diff --git a/contrib/fssim/testcases/daptest6.txt b/contrib/fssim/testcases/daptest6.txt new file mode 100644 index 00000000..bfb2f56f --- /dev/null +++ b/contrib/fssim/testcases/daptest6.txt @@ -0,0 +1,9 @@ +index q0 +touch a/b/c/foo +index q1 +touch a/b/c/foo +touch a/b/c +touch a/b +log +index q2 +exit diff --git a/contrib/fssim/testcases/fssimin-Log.txt b/contrib/fssim/testcases/fssimin-Log.txt new file mode 100644 index 00000000..9618232b --- /dev/null +++ b/contrib/fssim/testcases/fssimin-Log.txt @@ -0,0 +1,14 @@ +index l0 +touch A/D/bar +touch A/B/C/foo +echo "Hello world!" > A/B/C/foo +cd /A/B/C +cp foo / +cd +rm A/B/C/foo +rm foo +echo "foo!" > foo +log +log -t +index l1 +exit diff --git a/contrib/fssim/testcases/fssimin-TI.txt b/contrib/fssim/testcases/fssimin-TI.txt new file mode 100644 index 00000000..9998490f --- /dev/null +++ b/contrib/fssim/testcases/fssimin-TI.txt @@ -0,0 +1,77 @@ +index J0 +touch A1/B1/C1/c +echo "some " > A1/B1/b +md A1/B2 A1/B3 +tree +index J1 +#Case 1. Deleting a file +# step 1: delete /A1/B1/b +rm A1/B1/b +tree +index J2 +#Case 2. Adding data to a file, and rename the file +# step 1: adding data at end of existing file /A1/B1/b +# step 2: rename the file b to b.NEW +echo data >> A1/B1/b +mv A1/B1/b A1/B1/b.NEW +tree +index J3 +#Case 3: 1. Deleting a file (file UID: XXX), +# and create a new file with same name (file UID: YYY) +touch foo +ls +rm foo +touch foo +ls +index J4 +rm foo +#Case 4. Creating a new file with same name, after renaming the original one +# step 1: rename file b to b.bak +# step 2: create a new file as b +touch b +mv b b.bak +touch b +ls +index J5 +#Case 5. Moving whole subdirectory to another location +# step 1: move directory B1 under directory B2 +tree +mv A1/B1 A1/B2/B1 +tree +index J6 +#Case 6. Moving a file to other location, following by moving its parent +# directory to different location (or move a subdirectory first, and then +# move a file inside the subdirectory to other location). +# step 1: move file /A1/B1/C1/c to B2 (now it becomes /A1/B2/c) +# step 2: move directory B1 (not including c) under B3 +# or, +# step 1: move directory B1 under B3 +# step 2: move file c in /A1/B3/B1/C1 directory to B2 +md A1/B1/C1 A1/B3 +touch A1/B1/C1/c +tree +index XY +mv A1/B1/C1/c A1/B2 +mv A1/B1 A1/B3 +tree -o +log +index J7 +#Case 7. Moving a file more than once +# step 1: move file b to directory B2 +# step 2: move the same file (/A1/B2/b) to B3 +mv b /A1/B2 +ll +ls /A1/B2 +mv /A1/B2/b A1/B3 +ll /A1/B2 +ls /A1/B3 +tree +index J8 +#Case 8. Create a file and delete it within an interval +# step 1: delete file c +# step 2: create file c in /A1/B1/C +touch c +rm c +ll +index J9 +exit diff --git a/contrib/fssim/testcases/fssimin10.txt b/contrib/fssim/testcases/fssimin10.txt new file mode 100644 index 00000000..13902adc --- /dev/null +++ b/contrib/fssim/testcases/fssimin10.txt @@ -0,0 +1,63 @@ +index -f t10-0 +# /A +# |_ /B +# | |_ n1 +# |_ n2 +# |_ n3 +# /C +# |_ n4 +# |_ n5 +# /n1 +# /D +# /E +# +touch /A/B/n1 /A/n2 /A/n3 /C/n4 /C/n5 n1 +mkdir D E +tree +index -i t10-1 +fsck +# 1. Delete /A/B/n1 +# Rename the directory /C to /A/B/n1 +rm /A/B/n1 +mv /C /A/B/n1 +tree -o +index -i t10-2 +fsck +# 2. Append data data to /A/B/n1 +# Delete /A/B/n1 +# Rename file /n1 to /A/B/n1 +touch /A/B/n1 +rm /A/B/n1 +mv n1 /A/B/n1 +tree -o +index -i t10-3 +fsck +# 3. Delete /E +# Rename /D to /E +rd E +mv D E +tree +index -i t10-4 +fsck +# 4. Append data to /A/B/n1 +# Rename /A/B/n1 to /A/n1 +# Delete /A/B +# Rename /C to /A/B +# Move back /A/n1 to the new directory under /A/B +md C +touch /A/B/n1 +mv A/B/n1 A/n1 +rd A/B +mv C /A/B +mv n1 /A/B +tree +log +index t10-5 +fsck +# 5. Delete /n1 +# Create a new non-empty file /n1 +# Delete /n1 again +# Create an empty file /n1 + +exit + diff --git a/contrib/fssim/testcases/fssimin4.txt b/contrib/fssim/testcases/fssimin4.txt new file mode 100644 index 00000000..aa5cc5f5 --- /dev/null +++ b/contrib/fssim/testcases/fssimin4.txt @@ -0,0 +1,7 @@ +index t4-0 +touch asdf/foo +rm asdf/foo +rd asdf +index t4-1 +exit + diff --git a/contrib/fssim/testcases/fssimin5.txt b/contrib/fssim/testcases/fssimin5.txt new file mode 100644 index 00000000..036efda7 --- /dev/null +++ b/contrib/fssim/testcases/fssimin5.txt @@ -0,0 +1,8 @@ +index t5-0 +touch asdf/qwer/zxcv/foo +index t5-1 +rm asdf/qwer/zxcv/foo +ll asdf/qwer/zxcv +index t5-2 +ll asdf/qwer/zxcv +exit diff --git a/contrib/fssim/testcases/fssimin6.txt b/contrib/fssim/testcases/fssimin6.txt new file mode 100644 index 00000000..761dc985 --- /dev/null +++ b/contrib/fssim/testcases/fssimin6.txt @@ -0,0 +1,8 @@ +index t6-0 +md Dir1 +touch Dir2/Dir3/foo Dir2/Dir3/bar +index t6-1 +touch Dir2/Dir3/baz +rd Dir1 +index t6-2 +exit diff --git a/contrib/fssim/testcases/fssimin7.txt b/contrib/fssim/testcases/fssimin7.txt new file mode 100644 index 00000000..0638fba9 --- /dev/null +++ b/contrib/fssim/testcases/fssimin7.txt @@ -0,0 +1,35 @@ +index t7-0 +touch asdf/foo asdf/bar +rm asdf/foo +rm asdf/bar +# asdf has deleted files foo and bar +ll asdf +rd asdf +ll +# asdf is deleted +md asdf +ll +# asdf is recreated, has a different oid (so is different directory) +ll asdf +rd asdf +# asdf is deleted again +ll +touch qwer/baz qwer/bar +# qwer is created, contains only bar and baz +ll +ll qwer +tree -o +mv qwer asdf +# qwer (with bar and baz) is moved to deleted asdf, asdf has new oid +ll +# qwer is now deleted, asdf is recreated (share same oid) +ll asdf +# asdf should contain bar, baz +md qwer +# qwer is recreated, gets new oid so is different directory +ll +ll asdf +ll qwer +tree -o +index t7-1 +exit diff --git a/contrib/fssim/testcases/fssimin8.txt b/contrib/fssim/testcases/fssimin8.txt new file mode 100644 index 00000000..f438efdb --- /dev/null +++ b/contrib/fssim/testcases/fssimin8.txt @@ -0,0 +1,18 @@ +index t8-0 +touch asdf/foo asdf/bar +index t8-1 +# index t8-1 contains asdf with both foo and bar +rm asdf/foo +rm asdf/bar +rd asdf +# remove foo and bar, and directory asdf +touch asdf +#recreate asdf as a file +rm asdf +# now remove asdf +md asdf +# and recreate asdf as a directory again +index t8-2 +tree +exit + diff --git a/contrib/fssim/testcases/fssimin9.txt b/contrib/fssim/testcases/fssimin9.txt new file mode 100644 index 00000000..a55f2638 --- /dev/null +++ b/contrib/fssim/testcases/fssimin9.txt @@ -0,0 +1,17 @@ +index t9-0 +md qwer +touch asdf/zxcv/uiop/foo asdf/zxcv/uiop/bar asdf/zxcv/hjkl/baz +tree -o +# directory is qwer empty, adsf contains zxcv with subdirs and files +index t9-1 +mv asdf/zxcv qwer +tree -o +ll asdf +# move entire zxcv subtree to qwer, verify everything moves correctly +index t9-2 +md asdf/zxcv +cp -r qwer/zxcv asdf/zxcv +tree -o +index t9-3 +exit + diff --git a/contrib/fssim/testcases/logerr.txt b/contrib/fssim/testcases/logerr.txt new file mode 100644 index 00000000..df776b5d --- /dev/null +++ b/contrib/fssim/testcases/logerr.txt @@ -0,0 +1,19 @@ +touch A/B/foo +touch A/B/bar +touch A/B/baz +touch A/B/C/foo +touch A/B/C/bar +touch A/B/C/baz +log +index l0 +rm A/B/foo +rm A/B/bar +rm A/B/baz +rm A/B/C/foo +rm A/B/C/bar +rm A/B/C/baz +rd A/B/C +rd A/B +log +index l1 +exit diff --git a/contrib/fssim/testcases/tc001.txt b/contrib/fssim/testcases/tc001.txt new file mode 100644 index 00000000..2360faca --- /dev/null +++ b/contrib/fssim/testcases/tc001.txt @@ -0,0 +1,14 @@ +mkdir /DA1 +mkdir /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9/DB10 +index initial_tc001 +mv /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9/DB10 /DA1 +mv /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9 /DA1/DB10 +mv /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8 /DA1/DB10/DB9 +mv /DB1/DB2/DB3/DB4/DB5/DB6/DB7 /DA1/DB10/DB9/DB8 +mv /DB1/DB2/DB3/DB4/DB5/DB6 /DA1/DB10/DB9/DB8/DB7 +mv /DB1/DB2/DB3/DB4/DB5 /DA1/DB10/DB9/DB8/DB7/DB6 +mv /DB1/DB2/DB3/DB4 /DA1/DB10/DB9/DB8/DB7/DB6/DB5 +mv /DB1/DB2/DB3 /DA1/DB10/DB9/DB8/DB7/DB6/DB5/DB4 +mv /DB1/DB2 /DA1/DB10/DB9/DB8/DB7/DB6/DB5/DB4/DB3 +index flipped_tc001 +exit diff --git a/contrib/fssim/testcases/tc002.txt b/contrib/fssim/testcases/tc002.txt new file mode 100644 index 00000000..a901c792 --- /dev/null +++ b/contrib/fssim/testcases/tc002.txt @@ -0,0 +1,8 @@ +mkdir /DB0 +mkdir /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9/DB10 +index initial_tc002 +mv /DB1/DB2 /DB0 +mv /DB0/DB2/DB3 /DB1 +mv /DB1/DB3/DB4 /DB0/DB2 +index shuffle_tc002 +exit diff --git a/contrib/fssim/testcases/tc003.txt b/contrib/fssim/testcases/tc003.txt new file mode 100644 index 00000000..e83016ee --- /dev/null +++ b/contrib/fssim/testcases/tc003.txt @@ -0,0 +1,29 @@ +mkdir /DB0 +mkdir /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9/DB10 +tree +index initial_tc003 +mv /DB1/DB2 /DB0 +mv /DB0/DB2/DB3 /DB1 +mv /DB1/DB3/DB4 /DB0/DB2 +mv /DB0/DB2/DB4/DB5 /DB1/DB3 +mv /DB1/DB3/DB5/DB6 /DB0/DB2/DB4 +mv /DB0/DB2/DB4/DB6/DB7 /DB1/DB3/DB5 +mv /DB1/DB3/DB5/DB7/DB8 /DB0/DB2/DB4/DB6 +mv /DB0/DB2/DB4/DB6/DB8/DB9 /DB1/DB3/DB5/DB7 +mv /DB1/DB3/DB5/DB7/DB9/DB10 /DB0/DB2/DB4/DB6/DB8 +tree -o +mv /DB0 /DA1 +mv /DA1/DB2 /DA1/DA2 +mv /DA1/DA2/DB4 /DA1/DA2/DA3 +mv /DA1/DA2/DA3/DB6 /DA1/DA2/DA3/DA4 +mv /DA1/DA2/DA3/DA4/DB8 /DA1/DA2/DA3/DA4/DA5 +mv /DA1/DA2/DA3/DA4/DA5/DB10 /DA1/DA2/DA3/DA4/DA5/DA6 +mv /DB1/DB3/DB5/DB7/DB9 /DB1/DB3/DB5/DB7/DB1 +mv /DB1/DB3/DB5/DB7 /DB1/DB3/DB5/DB2 +mv /DB1/DB3/DB5 /DB1/DB3/DB3 +mv /DB1/DB3 /DB1/DB4 +mv /DB1 /DB5 +tree -o +#log +index shuffle_tc003 +exit diff --git a/contrib/fssim/testcases/tc004.txt b/contrib/fssim/testcases/tc004.txt new file mode 100644 index 00000000..de81eaab --- /dev/null +++ b/contrib/fssim/testcases/tc004.txt @@ -0,0 +1,34 @@ +mkdir /DB0 +mkdir /DB1/DB2/DB3/DB4/DB5/DB6/DB7/DB8/DB9/DB10 +index initial_tc004 +mv /DB1/DB2 /DB0 +mv /DB0/DB2/DB3 /DB1 +mv /DB1/DB3/DB4 /DB0/DB2 +mv /DB0/DB2/DB4/DB5 /DB1/DB3 +mv /DB1/DB3/DB5/DB6 /DB0/DB2/DB4 +mv /DB0/DB2/DB4/DB6/DB7 /DB1/DB3/DB5 +mv /DB1/DB3/DB5/DB7/DB8 /DB0/DB2/DB4/DB6 +mv /DB0/DB2/DB4/DB6/DB8/DB9 /DB1/DB3/DB5/DB7 +mv /DB1/DB3/DB5/DB7/DB9/DB10 /DB0/DB2/DB4/DB6/DB8 +mv /DB0 /DA1 +mv /DA1/DB2 /DA1/DA2 +mv /DA1/DA2/DB4 /DA1/DA2/DA3 +mv /DA1/DA2/DA3/DB6 /DA1/DA2/DA3/DA4 +mv /DA1/DA2/DA3/DA4/DB8 /DA1/DA2/DA3/DA4/DA5 +mv /DA1/DA2/DA3/DA4/DA5/DB10 /DA1/DA2/DA3/DA4/DA5/DA6 +mv /DB1/DB3/DB5/DB7/DB9 /DB1/DB3/DB5/DB7/DB1 +mv /DB1/DB3/DB5/DB7 /DB1/DB3/DB5/DB2 +mv /DB1/DB3/DB5 /DB1/DB3/DB3 +mv /DB1/DB3 /DB1/DB4 +mv /DB1 /DB5 +tree +rmdir /DA1/DA2/DA3/DA4/DA5/DA6 +rmdir /DA1/DA2/DA3/DA4/DA5 +rmdir /DA1/DA2/DA3/DA4 +rmdir /DA1/DA2/DA3 +rmdir /DB5/DB4/DB3/DB2/DB1 +rmdir /DB5/DB4/DB3/DB2 +rmdir /DB5/DB4/DB3 +tree +index shuffle_remove_tc004 +exit diff --git a/contrib/fssim/testcases/tc005.txt b/contrib/fssim/testcases/tc005.txt new file mode 100644 index 00000000..04620dad --- /dev/null +++ b/contrib/fssim/testcases/tc005.txt @@ -0,0 +1,13 @@ +mkdir /DA0 +mkdir /DB0 +touch /DA0/fa0 +touch /DB0/fb0 +tree +index initial_tc005 +rm /DA0/fa0 +touch /DA0/fa0 +mv /DB0/fb0 /DB0/fb1 +mv /DB0/fb1 /DB0/fb0 +tree +index new_samename_tc005 +exit diff --git a/contrib/ut-incindex/README.md b/contrib/ut-incindex/README.md new file mode 100644 index 00000000..ec3d9658 --- /dev/null +++ b/contrib/ut-incindex/README.md @@ -0,0 +1,13 @@ +# Unit Tests for incremental index + +This is a directory for having unit tests for incremental index feature. + +## How to run + +### Basic operation test + + 1. `cd` to this directory + 2. Run the basic test with `./ut-basic.sh [mount_point]` + - The test script formats a (filebackend) tape under `/tmp/ltfstape`, start ltfs and stop ltfs automatically. + - If `[mount_point]` is not specified, the script uses `/tmp/mnt` + - You can pass the specific `ltfs` binary directory with teh environmental value `LTFS_BIN_PATH` diff --git a/contrib/ut-incindex/ut-basic.sh b/contrib/ut-incindex/ut-basic.sh new file mode 100755 index 00000000..de61a29b --- /dev/null +++ b/contrib/ut-incindex/ut-basic.sh @@ -0,0 +1,96 @@ +#!/bin/sh + +source ./utils.sh + +MOUNTPOINT='/tmp/mnt' +TAPE_PATH='/tmp/ltfstape' + +if [ $# == 1 ]; then + MOUNTPOINT=$1 +elif [ $# -gt 2 ]; then + MOUNTPOINT=$1 + TAPE_PATH=$2 +fi + +# Format LTFS +FormatLTFS ${TAPE_PATH} +if [ $? != 0 ]; then + exit 1 +fi + +# Launch LTFS +LaunchLTFS ${MOUNTPOINT} ${TAPE_PATH} +if [ $? != 0 ]; then + exit 1 +fi + +# 1. CREATE DIRS +# Create a few dirs and files but all objects are handles by new dirs +echo "1. CREATE DIRS" +mkdir -p ${MOUNTPOINT}/dir1/dir11 +mkdir -p ${MOUNTPOINT}/dir1/dir12 +mkdir -p ${MOUNTPOINT}/dir2/dir21 +mkdir -p ${MOUNTPOINT}/dir2/dir22 +echo "AAA" > ${MOUNTPOINT}/dir1/file11 +echo "AAA" > ${MOUNTPOINT}/dir1/dir11/file111 +echo "AAA" > ${MOUNTPOINT}/dir1/dir11/file112 +IncrementalSync ${MOUNTPOINT} '1.CREATE_DIRS' +if [ $? != 0 ]; then + exit 1 +fi + +# 2. CREATE FILES +# Create files for checking file creation and directory traverse +echo "2. CREATE FILES" +echo "AAA" > ${MOUNTPOINT}/dir1/dir11/file113 +echo "AAA" > ${MOUNTPOINT}/dir1/dir12/file121 +echo "AAA" > ${MOUNTPOINT}/dir2/dir22/file221 +echo "AAA" > ${MOUNTPOINT}/dir2/file21 +IncrementalSync ${MOUNTPOINT} '2.CREATE_FILES' +if [ $? != 0 ]; then + exit 1 +fi + +# 3. MODIFY FILES +# Modify contents of files. Need to check /dir2 doesn't have meta-data on the incremental index +echo "3. MODIFY FILES" +echo "BBB" > ${MOUNTPOINT}/dir1/dir11/file111 +echo "BBB" > ${MOUNTPOINT}/dir2/dir22/file221 +echo "BBB" > ${MOUNTPOINT}/dir1/file11 +IncrementalSync ${MOUNTPOINT} '3.MODIFY_FILES' +if [ $? != 0 ]; then + exit 1 +fi + +# 4. MODIFY DIRS +# Modify directory's meta-data. Need to check both /dir1 and /dir1/dir11 has meta-data +# on the incremental index +echo "4. MODIFY DIRS" +AddXattr ${MOUNTPOINT}/dir1 'ut-attr1' 'val1' +echo "CCC" > ${MOUNTPOINT}/dir1/dir11/file111 +IncrementalSync ${MOUNTPOINT} '4.MODIFY_DIRS' +if [ $? != 0 ]; then + exit 1 +fi + +# 5. DELETE FILES +echo "5. DELETE FILES" +rm ${MOUNTPOINT}/dir1/dir11/* +IncrementalSync ${MOUNTPOINT} '5.DELETE_FILES' +if [ $? != 0 ]; then + exit 1 +fi + +# 6. DELETE DIR +echo "5. DELETE DIR" +rm -rf ${MOUNTPOINT}/dir1/dir11 +IncrementalSync ${MOUNTPOINT} '6.DELETE_DIR' +if [ $? != 0 ]; then + exit 1 +fi + +# Stop LTFS +StopLTFS ${MOUNTPOINT} ${TAPE_PATH} +if [ $? != 0 ]; then + exit 1 +fi diff --git a/contrib/ut-incindex/utils.sh b/contrib/ut-incindex/utils.sh new file mode 100755 index 00000000..ca79d3c6 --- /dev/null +++ b/contrib/ut-incindex/utils.sh @@ -0,0 +1,197 @@ +#!/bin/sh + +PLATFORM=`uname` +ECHO='/bin/echo' + +if [ "x${LTFS_BIN_PATH}" == 'x' ]; then + LTFS_BIN_PATH='/usr/local/bin' +fi + +AddXattr() +{ + if [ "x$1" == "x" ]; then + "Need to a target to set xattr" + return 1 + else + TARGET=$1 + fi + + if [ "x$2" == "x" ]; then + "Need to a name to set xattr" + return 1 + else + NAME=$2 + fi + + if [ "x$3" == "x" ]; then + "Need to a value to set xattr" + return 1 + else + VAL=$2 + fi + + ${ECHO} -n "Setting a xattr ${NAME} to ${TARGET} ... " + if [ "$PLATFORM" == "Darwin" ]; then + /usr/bin/xattr -w ${NAME} ${VAL} ${TARGET} + else + /usr/bin/attr -s ${NAME} -V ${VAL} ${TARGET} + fi + + if [ $? == 0 ]; then + ${ECHO} "Done" + return 0 + else + ${ECHO} "Failed" + return 1 + fi + +} + +IncrementalSync() +{ + if [ "x$1" == "x" ]; then + MOUNTPOINT='/tmp/mnt' + else + MOUNTPOINT=$1 + fi + + if [ "x$2" == "x" ]; then + MSG='Incremental Sync' + else + MSG=$2 + fi + + ${ECHO} -n "Syncing LTFS (Incremental) ... " + if [ "$PLATFORM" == "Darwin" ]; then + /usr/bin/xattr -w ltfs.vendor.IBM.IncrementalSync ${MSG} ${MOUNTPOINT} + else + /usr/bin/attr -s ltfs.vendor.IBM.IncrementalSync -V ${MSG} ${MOUNTPOINT} + fi + + if [ $? == 0 ]; then + ${ECHO} "Done" + return 0 + else + ${ECHO} "Failed" + return 1 + fi +} + +FullSync() +{ + if [ "x$1" == "x" ]; then + MOUNTPOINT='/tmp/mnt' + else + MOUNTPOINT=$1 + fi + + if [ "x$2" == "x" ]; then + MSG='Full Sync' + else + MSG=$2 + fi + + ${ECHO} "Syncing LTFS (Full) ... " + if [ "$PLATFORM" == "Darwin" ]; then + /usr/bin/xattr -w ltfs.vendor.IBM.FullSync ${MSG} ${MOUNTPOINT} + else + /usr/bin/attr -s ltfs.vendor.IBM.FullSync -V ${MSG} ${MOUNTPOINT} + fi + + if [ $? == 0 ]; then + ${ECHO} "Done" + return 0 + else + ${ECHO} "Failed" + return 1 + fi +} + +FormatLTFS() +{ + if [ "x$1" == "x" ]; then + TAPE_PATH='/tmp/ltfstape' + else + TAPE_PATH=$1 + fi + + if [ ! -d ${TAPE_PATH} ]; then + ${ECHO} "Creating tape directory for file backend: ${TAPE_PATH}" + mkdir -p ${TAPE_PATH} + if [ $? != 0 ]; then + ${ECHO} "Failed to create a tape path: ${TAPE_PATH}" + return 1 + fi + fi + + ${ECHO} "Formatting tape directory with the file backend on ${TAPE_PATH} ... " + ${LTFS_BIN_PATH}/mkltfs -f -e file -d ${TAPE_PATH} + if [ $? != 0 ]; then + ${ECHO} "Failed to format a tape path: ${TAPE_PATH}" + return 1 + fi + + ${ECHO} "Formatted the file backend on ${TAPE_PATH}" + return 0 +} + +LaunchLTFS() +{ + if [ "x$1" == "x" ]; then + MOUNTPOINT='/tmp/mnt' + else + MOUNTPOINT=$1 + fi + + if [ "x$2" == "x" ]; then + TAPE_PATH='/tmp/ltfstape' + else + TAPE_PATH=$2 + fi + + if [ ! -d ${MOUNTPOINT} ]; then + ${ECHO} "Creating mount point for LTFS: ${MOUNTPOINT}" + mkdir -p ${MOUNTPOINT} + if [ $? != 0 ]; then + ${ECHO} "Failed to create a mount point" + return 1 + fi + fi + + if [ ! -d ${TAPE_PATH} ]; then + ${ECHO} "Creating tape directory for file backend: ${TAPE_PATH}" + mkdir -p ${TAPE_PATH} + if [ $? != 0 ]; then + ${ECHO} "Failed to create a tape path: ${TAPE_PATH}" + return 1 + fi + + ${ECHO} "Formatting tape directory with the file backend" + ${LTFS_BIN_PATH}/mkltfs -f -e file -d ${TAPE_PATH} + if [ $? != 0 ]; then + ${ECHO} "Failed to format a tape path: ${TAPE_PATH}" + return 1 + fi + fi + + ${ECHO} "Launching LTFS with the file backend" + ${LTFS_BIN_PATH}/ltfs -o tape_backend=file -o sync_type=unmount -o devname=${TAPE_PATH} ${MOUNTPOINT} + if [ $? != 0 ]; then + ${ECHO} "Failed to launch LTFS on ${MOUNTPOINT}" + return 1 + fi + + ${ECHO} "LTFS is launched on ${MOUNTPOINT}" + return 0 +} + +StopLTFS() +{ + if [ "x$1" == "x" ]; then + MOUNTPOINT='/tmp/mnt' + else + MOUNTPOINT=$1 + fi + + sudo umount ${MOUNTPOINT} +} diff --git a/man/Makefile.am b/man/Makefile.am index f18e081d..f6eec413 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -1,4 +1,4 @@ -man_MANS = mkltfs.8 ltfsck.8 ltfs-sde.8 ltfs_ordered_copy.1 +man_MANS = mkltfs.8 ltfsck.8 ltfs.8 ltfs_ordered_copy.1 ltfsindextool.8 man-clean: rm $(man_MANS) @@ -6,6 +6,7 @@ man-clean: man-rebuild: docbook2man sgml/mkltfs.sgml docbook2man sgml/ltfsck.sgml - docbook2man sgml/ltfs-sde.sgml + docbook2man sgml/ltfs.sgml docbook2man sgml/ltfs_ordered_copy.sgml + docbook2man sgml/ltfsindextool.sgml rm -f manpage.* diff --git a/man/ltfs-sde.8 b/man/ltfs.8 similarity index 88% rename from man/ltfs-sde.8 rename to man/ltfs.8 index af058dfe..7430edf3 100644 --- a/man/ltfs-sde.8 +++ b/man/ltfs.8 @@ -70,8 +70,11 @@ Only use for a CM corrupted medium \fB-o device_list\fR Show available tape devices .TP -\fB-o rollback_mount=\fIgen\fB\fR -Attempt to mount on previous index generation (read-only mount) +\fB-o rollback_mount=\fIgen|index_file\fB\fR +Attempt to mount on previous index generation on tape or specified index file by read-only mode + +When both index_file and device name is specified on -o devname option, normal read-only mount is attempted. +When index_file is only specified, meta-data read-only mode is attempted. EAGAIN would be returned at accessing contents of file. .TP \fB-o release_device\fR Clear device reservation (should be specified with -o devname @@ -179,8 +182,10 @@ Override output verbosity directly (default: 2) \fB-o noeject\fR Do not eject the cartridge after unmount (default) .TP -\fB-o capture_index\fR -Capture latest index to work directory at unmount +\fB-o capture_index=\fIdir\fB\fR +Capture index to the specified directory by dir when index is updated. +File name of each index is [BARCODE]-[GEN]-[PARTITION].xml if tape serial (barcode) +is specified at format time. Otherwise it is [VOL_UUID]-[GEN]-[PARTITION].xml. .TP \fB-o scsi_append_only_mode=\fIon|off\fB\fR Set the tape device append-only mode (default=on) diff --git a/man/ltfs_ordered_copy.1 b/man/ltfs_ordered_copy.1 index 0444daee..b16b2d51 100644 --- a/man/ltfs_ordered_copy.1 +++ b/man/ltfs_ordered_copy.1 @@ -4,7 +4,7 @@ ltfs_ordered_copy \- Copy files from source to destination with LTFS order optimization .SH SYNOPSIS .sp -\fBltfs_ordered_copy\fR [ \fB-p\fR ] [ \fB-r\fR ] [ \fB-t \fITARGET_DIRECTORY\fB\fR ] [ \fB--keep-tree \fICUTOFF_PREFIX\fB\fR ] [ \fB-a\fR ] [ \fB-v\fR ] [ \fB--verbose \fILOG_LEVEL\fB\fR ] [ \fB-q\fR ] [ \fB-h\fR ] [ \fBSOURCE \fR\fI...\fR ] [ \fBDESTINATION\fR ] +\fBltfs_ordered_copy\fR [ \fB-p\fR ] [ \fB-r\fR ] [ \fB-t \fITARGET_DIRECTORY\fB\fR ] [ \fB--keep-tree \fICUTOFF_PREFIX\fB\fR ] [ \fB-a\fR ] [ \fB-v\fR ] [ \fB--verbose \fILOG_LEVEL\fB\fR ] [ \fB-q\fR ] [ \fB-h\fR ] [ \fBSOURCE \fR\fI...\fR ] [ \fBDESTINATION\fR ] .SH "DESCRIPTION" .PP \fBltfs_ordered_copy\fR is a program to copy files from source to destination with LTFS diff --git a/man/ltfsck.8 b/man/ltfsck.8 index 9e0c6cae..aa024f34 100644 --- a/man/ltfsck.8 +++ b/man/ltfsck.8 @@ -90,12 +90,13 @@ Use the specified key manager interface backend (default: none) \fB-x, --fulltrace\fR Enable full function call tracing (slow) .TP -\fB--capture-index\fR -Capture index information to the current directory -(-g is effective for this option) +\fB--capture-index=\fIdir\fB\fR +Capture indexes read successfully to the specified directory by dir. (-g is effective for this option) +File name of each index is [BARCODE]-[GEN]-[PARTITION].xml if tape serial (barcode) +is specified at format time. Otherwise it is [VOL_UUID]-[GEN]-[PARTITION].xml. .TP \fB--salvage-rollback-points\fR List the rollback points of the cartridge that has no EOD .SH "SEE ALSO" .PP -ltfs-sde(8), mkltfs(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). +ltfs(8), mkltfs(8), ltfsindextool(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). diff --git a/man/ltfsindextool.8 b/man/ltfsindextool.8 new file mode 100644 index 00000000..dd932c29 --- /dev/null +++ b/man/ltfsindextool.8 @@ -0,0 +1,72 @@ +.\" auto-generated by docbook2man-spec from docbook-utils package +.TH "LTFSINDEXTOOL" "8" "19 August 2021" "LTFS" "LTFS Command Reference" +.SH NAME +ltfsindextool \- Low level index checking tool for LTFS +.SH SYNOPSIS +.sp +\fBltfsindextool\fR [ \fB-d \fIname\fB\fR ] [ \fB-p \fIpart_num\fB\fR ] [ \fB-s \fIblock\fB\fR ] [ \fB-b \fInum\fB\fR ] [ \fB-i \fIfile\fB\fR ] [ \fB-e \fIname\fB\fR ] [ \fB-q\fR ] [ \fB-t\fR ] [ \fB-V\fR ] [ \fB-h\fR ] [ \fB-p\fR ] [ \fB\fIfilename\fB\fR ] +.SH "DESCRIPTION" +.PP +\fBltfsindextool\fR is a low level index checking tool. +.PP +There are 2 features. One is captureing indexes on a tape, the other is checking captured index from a file. +The command runs as index checking mode when filename is specified. It runs as index capturing mode when +-d option is specified. It runs as index checking mode when both filename and -d option are specified. +-p, -s --output-dir, -b is available only when it runs under index captureing mode. They would be ignored +when it runs under index checking mode. +.SH "OPTIONS" +.PP +These programs follow the usual GNU command line syntax, +with long options starting with two dashes ('-'). A summary of +options is included below. For a complete description, see the +\fBInfo\fR files. +.TP +\fB-d, --device=\fIname\fB\fR +Tape device name to capture indexes. On Linux, \fIname\fR is like +\&'/dev/IBMtape0', on OSX, \fIname\fR is like '0'. +.TP +\fB-p, --partition=\fIpart_num\fB\fR +Partition to capture indexes. Shall be 0 or 1. Capture indexes on both partitions +.TP +\fB-s, --start-pos=\fIblock\fB\fR +Block number to start capturing indexes +.TP +\fB--output-dir=\fIdir\fB\fR +Directory to store captured indexes +.TP +\fB-b, --blocksize=\fInum\fB\fR +Specify the LTFS record size +.TP +\fB-i, --config=\fIname\fB\fR +Use the specified configuration file +.TP +\fB-e, --backend=\fIname\fB\fR +Use the specified tape device backend +.TP +\fB--kmi-backend=\fIname\fB\fR +Use the specified key manager interface backend (default: none) +.TP +\fB-q, --quiet\fR +Suppress progress information and general messages +.TP +\fB-t, --trace\fR +Enable function call tracing +.TP +\fB-V, --version\fR +Version information +.TP +\fB-h, --help\fR +Show help information +.SS "USAGE EXAMPLE" +.sp +.nf + ltfsindextool -d /dev/sg10 + ltfsindextool -d /dev/sg10 -p 1 --output-dir=/foo + ltfsindextool -d ltfs-index-1-35.xml + +.sp +.fi +.PP +.SH "SEE ALSO" +.PP +ltfs(8), mkltfs(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). diff --git a/man/mkltfs.8 b/man/mkltfs.8 index 4ae3262c..aea0c7ec 100644 --- a/man/mkltfs.8 +++ b/man/mkltfs.8 @@ -4,7 +4,7 @@ mkltfs \- Format a tape in the drive to LTFS format .SH SYNOPSIS .sp -\fBmkltfs\fR \fB-d \fIname\fB\fR [ \fB-f\fR ] [ \fB-s \fIid\fB\fR ] [ \fB-n \fIname\fB\fR ] [ \fB-r \fIrules\fB\fR ] [ \fB-w\fR ] [ \fB-q\fR ] [ \fB-t\fR ] [ \fB-V\fR ] [ \fB-h\fR ] [ \fB-p\fR ] +\fBmkltfs\fR \fB-d \fIname\fB\fR [ \fB-f\fR ] [ \fB-s \fIid\fB\fR ] [ \fB-n \fIname\fB\fR ] [ \fB-r \fIrules\fB\fR ] [ \fB-w\fR ] [ \fB-q\fR ] [ \fB-t\fR ] [ \fB-V\fR ] [ \fB-h\fR ] [ \fB-p\fR ] .SH "DESCRIPTION" .PP \fBmkltfs\fR is a program to format a media for use with @@ -75,7 +75,7 @@ Full help, including advanced options /home/piste/ltfs05-sde/bin/mkltfs --device=/dev/IBMtape0 --rules="size=100K" /home/piste/ltfs05-sde/bin/mkltfs --device=/dev/IBMtape0 --rules="size=1M/name=*.jpg" /home/piste/ltfs05-sde/bin/mkltfs --device=/dev/IBMtape0 --rules="size=1M/name=*.jpg:*.png" - + .sp .fi .PP @@ -117,4 +117,4 @@ This operation takes over 3 hours. Once you start, you cannot interrupt it. Use destructive format/unformat. This operation takes longer time in the LTO9 drive or later because of the media optimization procedure. .SH "SEE ALSO" .PP -ltfs-sde(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). +ltfs(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). diff --git a/man/sgml/ltfs-sde.sgml b/man/sgml/ltfs.sgml similarity index 93% rename from man/sgml/ltfs-sde.sgml rename to man/sgml/ltfs.sgml index 3d68be21..18bb59b5 100644 --- a/man/sgml/ltfs-sde.sgml +++ b/man/sgml/ltfs.sgml @@ -1,5 +1,5 @@ + GNU"> ]> @@ -156,9 +156,12 @@ - + - Attempt to mount on previous index generation (read-only mount) + Attempt to mount on previous index generation on tape or specified index file by read-only mode + + When both index_file and device name is specified on -o devname option, normal read-only mount is attempted. + When index_file is only specified, meta-data read-only mode is attempted. EAGAIN would be returned at accessing contents of file. @@ -356,9 +359,13 @@ - + - Capture latest index to work directory at unmount + + Capture index to the specified directory by dir when index is updated. + File name of each index is [BARCODE]-[GEN]-[PARTITION].xml if tape serial (barcode) + is specified at format time. Otherwise it is [VOL_UUID]-[GEN]-[PARTITION].xml. + diff --git a/man/sgml/ltfsck.sgml b/man/sgml/ltfsck.sgml index a5bd69fd..7cff98eb 100644 --- a/man/sgml/ltfsck.sgml +++ b/man/sgml/ltfsck.sgml @@ -209,11 +209,12 @@ M - + - Capture index information to the current directory - (-g is effective for this option) + Capture indexes read successfully to the specified directory by dir. (-g is effective for this option) + File name of each index is [BARCODE]-[GEN]-[PARTITION].xml if tape serial (barcode) + is specified at format time. Otherwise it is [VOL_UUID]-[GEN]-[PARTITION].xml. @@ -228,7 +229,7 @@ M SEE ALSO - ltfs-sde(8), mkltfs(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). + ltfs(8), mkltfs(8), ltfsindextool(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). diff --git a/man/sgml/ltfsindextool.sgml b/man/sgml/ltfsindextool.sgml new file mode 100644 index 00000000..d4530105 --- /dev/null +++ b/man/sgml/ltfsindextool.sgml @@ -0,0 +1,159 @@ + + GNU"> + ]> + + + LTFS Command Reference + + + + + &dhcommand; + + 8 + LTFS + + + + &dhcommand; + Low level index checking tool for LTFS + + + + + &dhcommand; + + + + + + + + + + + + filename + + + + + DESCRIPTION + + &dhcommand; is a low level index checking tool. + + + There are 2 features. One is captureing indexes on a tape, the other is checking captured index from a file. + The command runs as index checking mode when filename is specified. It runs as index capturing mode when + -d option is specified. It runs as index checking mode when both filename and -d option are specified. + -p, -s --output-dir, -b is available only when it runs under index captureing mode. They would be ignored + when it runs under index checking mode. + + + + + OPTIONS + + These programs follow the usual &gnu; command line syntax, + with long options starting with two dashes ('-'). A summary of + options is included below. For a complete description, see the + Info files. + + + + + + + + Tape device name to capture indexes. On Linux, name is like + '/dev/IBMtape0', on OSX, name is like '0'. + + + + + + + Partition to capture indexes. Shall be 0 or 1. Capture indexes on both partitions + + + + + + Block number to start capturing indexes + + + + + + Directory to store captured indexes + + + + + + Specify the LTFS record size + + + + + + Use the specified configuration file + + + + + + Use the specified tape device backend + + + + + + Use the specified key manager interface backend (default: none) + + + + + + Suppress progress information and general messages + + + + + + Enable function call tracing + + + + + + Version information + + + + + + Show help information + + + + + + USAGE EXAMPLE + + ltfsindextool -d /dev/sg10 + ltfsindextool -d /dev/sg10 -p 1 --output-dir=/foo + ltfsindextool -d ltfs-index-1-35.xml + + + + + + SEE ALSO + ltfs(8), mkltfs(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). + + + + + diff --git a/man/sgml/mkltfs.sgml b/man/sgml/mkltfs.sgml index 275dfc19..b5c99540 100644 --- a/man/sgml/mkltfs.sgml +++ b/man/sgml/mkltfs.sgml @@ -241,7 +241,7 @@ SEE ALSO - ltfs-sde(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). + ltfs(8), ltfsck(8), tape-backend(4), kmi-backend(4), ltfs.conf(5). diff --git a/messages/Makefile.am b/messages/Makefile.am index 3d712e00..26bc61dc 100644 --- a/messages/Makefile.am +++ b/messages/Makefile.am @@ -49,6 +49,7 @@ RESOURCES = \ libbin_mkltfs_dat.a \ libbin_ltfsck_dat.a \ + libbin_ltfsindextool_dat.a \ libbin_ltfs_dat.a \ libkmi_simple_dat.a \ libkmi_flatfile_dat.a \ diff --git a/messages/bin_ltfs/root.txt b/messages/bin_ltfs/root.txt index 3a9cd000..185fe1ca 100644 --- a/messages/bin_ltfs/root.txt +++ b/messages/bin_ltfs/root.txt @@ -3,7 +3,7 @@ // OO_Copyright_BEGIN // // -// Copyright 2010, 2020 IBM Corp. All rights reserved. +// Copyright 2010, 2022 IBM Corp. All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions @@ -52,7 +52,7 @@ root:table { //unused 14010E:string { "Cannot load I/O scheduler \'%s\'." } 14011E:string { "Cannot allocate LTFS volume structure." } 14012E:string { "Tape backend option parsing failed." } - 14013E:string { "Cannot mount the volume." } + 14013E:string { "Cannot mount the volume from %s." } //unused 14014E:string { "Cannot duplicate index criteria." } 14015W:string { "Volume does not allow index criteria override. Ignoring user-specified criteria." } 14016E:string { "Cannot format data placement rules (%d)." } @@ -210,7 +210,7 @@ root:table { // Reserved 14434E // Reserved 14435I 14436I:string { " -o device_list Show available tape devices" } - 14437I:string { " -o rollback_mount= Attempt to mount on previous index generation (read-only mount)" } + 14437I:string { " -o rollback_mount= Attempt to mount on previous index generation on tape or specified index file (read-only mount)" } // Reserved 14438I 14439I:string { " -o noeject Do not eject the cartridge after unmount (default)" } 14440I:string { " -o noatime Do not update index if only access times have changed (default)" } @@ -228,7 +228,7 @@ root:table { // Reserved 14451I // Reserved 14454I 14455I:string { " -o kmi_backend= Key manager interface implementation to use (default: %s, use \"none\" to disable)" } - 14456I:string { " -o capture_index Capture latest index to work directory at unmount" } + 14456I:string { " -o capture_index= Capture index to the specified directory by dir when index is updated" } // Reserved 14457I // Reserved 14458I // Reserved 14459I diff --git a/messages/bin_ltfsck/root.txt b/messages/bin_ltfsck/root.txt index 479fe514..02d8e98d 100644 --- a/messages/bin_ltfsck/root.txt +++ b/messages/bin_ltfsck/root.txt @@ -3,7 +3,7 @@ // OO_Copyright_BEGIN // // -// Copyright 2010, 2020 IBM Corp. All rights reserved. +// Copyright 2010, 2022 IBM Corp. All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions @@ -151,6 +151,10 @@ root:table { 16109E:string { "This operation is not allowed on this medium (%s)." } 16110E:string { "The --salvage-rollback-points option was specified against a normal cartridge." } 16111I:string { "The recovery process is skipped because of a locked cartridge (%d)." } + 16112W:string { "Cannot rename %s to %s (%d)." } + 16113W:string { "Cannot access to directory %s, disabled index capture mode (%d)." } + 16114I:string { "Index will be captured in %s." } + 16115I:string { "Index will not be captured." } // Help messages 16400I:string { "Usage: %s [options] filesys" } @@ -180,7 +184,7 @@ root:table { " (Must be used for a cartridge that cannot be recovered by a normal option.)" } 16422I:string { " -m, --full-index-info Display full index information (Effective only for -l option)" } 16423I:string { " --kmi-backend= Override the default key manager interface backend" } - 16424I:string { " --capture-index Capture index information to the current directory (-g is effective for this option)" } + 16424I:string { " --capture-index= Capture indexes to the specified directory (-g is effective for this option)" } 16425I:string { " --syslogtrace Enable diagnostic output to stderr and syslog" } 16426I:string { " -V, --version Version information" } 16427I:string { " --salvage-rollback-points List the rollback points of the cartridge that has no EOD" } diff --git a/messages/bin_ltfsindextool/en.txt b/messages/bin_ltfsindextool/en.txt new file mode 100644 index 00000000..44064bb4 --- /dev/null +++ b/messages/bin_ltfsindextool/en.txt @@ -0,0 +1,39 @@ +// +// +// OO_Copyright_BEGIN +// +// +// Copyright 2010, 2020 IBM Corp. All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// 2. Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// 3. Neither the name of the copyright holder nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +// POSSIBILITY OF SUCH DAMAGE. +// +// +// OO_Copyright_END +// + +en:table { + // This resource intentionally left blank. +} + diff --git a/messages/bin_ltfsindextool/en_US.txt b/messages/bin_ltfsindextool/en_US.txt new file mode 100644 index 00000000..ffd90dd2 --- /dev/null +++ b/messages/bin_ltfsindextool/en_US.txt @@ -0,0 +1,39 @@ +// +// +// OO_Copyright_BEGIN +// +// +// Copyright 2010, 2020 IBM Corp. All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// 2. Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// 3. Neither the name of the copyright holder nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +// POSSIBILITY OF SUCH DAMAGE. +// +// +// OO_Copyright_END +// + +en_US:table { + // This resource intentionally left blank. +} + diff --git a/messages/bin_ltfsindextool/root.txt b/messages/bin_ltfsindextool/root.txt new file mode 100644 index 00000000..b84dc906 --- /dev/null +++ b/messages/bin_ltfsindextool/root.txt @@ -0,0 +1,109 @@ +// +// +// OO_Copyright_BEGIN +// +// +// Copyright 2010, 2020 IBM Corp. All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// 2. Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// 3. Neither the name of the copyright holder nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +// POSSIBILITY OF SUCH DAMAGE. +// +// +// OO_Copyright_END +// + +// Messages for ltfsindextool. +root:table { + messages:table { + start_id:int { 19500 } + end_id: int { 19989 } + 19500I:string { "Starting ltfsindextool, %s version %s, log level %d." } + 19501E:string { "Cannot allocate LTFS volume structure." } + 19502I:string { "%s." } + 19503I:string { "GCC version is %s." } + 19504I:string { "Capture all indexes from both partitions." } + 19505I:string { "Capture all indexes from (%u, %llu)." } + 19506D:string { "Opening the device." } + 19507D:string { "Device opened." } + 19508E:string { "Cannot open backend \'%s\'." } + 19509E:string { "Cannot open key manager interface backend \'%s\'." } + 19510E:string { "Cannot open device \'%s\' (%d)." } + 19511E:string { "Could not initialize the key manager interface plug-in. \'%s\' (%d)." } + 19512E:string { "Key manager interface backend option parsing failed." } + 19513E:string { "Tape backend option parsing failed." } + 19514E:string { "Unknown option '%s %s'." } + 19515E:string { "Cannot set up tape device." } + 19516E:string { "Cannot allocate the read buffer." } + 19517E:string { "Cannot seek to the start position (%u, %llu, %d)." } + 19518E:string { "Cannot get the current position (%d)." } + 19519E:string { "Cannot read a block (%u, %llu, %ld)." } + 19520D:string { "Closing the device." } + 19521W:string { "Cannot unload backend." } + 19522D:string { "Device closed." } + 19523I:string { "Failed to capture index (%d)." } + 19524I:string { "Captured indexes successfully." } + 19525D:string { "Validating command line options." } + 19526E:string { "File name to check or device name to capture must be specified." } + 19527D:string { "Command line options are valid." } + 19528W:string { "Cannot unload key manager interface backend." } + 19529I:string { "Reading an index from (%u, %llu)." } + 19530I:string { "Block in (%u, %llu) does not look an index, seek to next position (%d)." } + 19531E:string { "Cannot seek to the next position (%u, %llu, %d)." } + 19532E:string { "Cannot allocate the file name." } + 19533E:string { "Cannot open the file %s (%d)." } + 19534I:string { "Reached to EOD (%u, %llu)." } + 19535I:string { "Reached to EOD but cannot get the position (%d)." } + 19536E:string { "Cannot write a block (%ld, %d)." } + 19537I:string { "Detected the end of the index (%u, %llu)." } + 19538E:string { "Detected the EOD in the middle of index (%u, %llu)." } + 19539I:string { "Wrote an index, length is %llu." } + 19540E:string { "Partition number must be 0 or 1." } + 19541E:string { "Operation mode is wrong." } + 19542I:string { "Launched by \"%s\"." } + 19543I:string { "Checking the index file \"%s\"." } + 19544I:string { "Checked the index successfully." } + 19545E:string { "Failed to check the index (%d)." } + 19546I:string { "%s version %s." } + 19547I:string { "Creating an index file: %s" } + 19548E:string { "Start position shall be 5 or larger (%llu)." } + + // Help messages + 19900I:string { "Usage: %s [options] [filename]" } + 19901I:string { "Available options are:" } + 19902I:string { " -d, --device= Tape device (Capture index to specified file when this option is specified. Otherwise run check process with specified file)" } + 19903I:string { " -p, --partition= Partition number 0 or 1, capture both partitions if not specified" } + 19904I:string { " -s, --start-pos= Block number to start capturing (default: %d)" } + 19905I:string { " --output-dir= Directory to store captured indexes (default: \'%s\')" } + 19906I:string { " -b, --blocksize= Specify the LTFS record size (default: %d)" } + 19907I:string { " -i, --config= Use the specified configuration file (default: %s)" } + 19908I:string { " -e, --backend= Use the specified tape device backend (default: %s)" } + 19909I:string { " --kmi-backend= Use the specified key manager interface backend (default: %s)" } + 19910I:string { " -q, --quiet Suppress progress information and general messages" } + 19911I:string { " -t, --trace Enable function call tracing" } + 19912I:string { " -V, --version Version information" } + 19913I:string { " -h, --help This help" } + 19914I:string { "Usage example:" } + 19915I:string { " %s --device=%s -p=%d" } + } +} diff --git a/messages/libltfs/root.txt b/messages/libltfs/root.txt index 646a4d95..48c112e9 100644 --- a/messages/libltfs/root.txt +++ b/messages/libltfs/root.txt @@ -3,7 +3,7 @@ // OO_Copyright_BEGIN // // -// Copyright 2010, 2020 IBM Corp. All rights reserved. +// Copyright 2010, 2022 IBM Corp. All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions @@ -97,7 +97,7 @@ root:table { 11002E:string { "Cannot instantiate LTFS volume: failed to allocate index data." } 11003E:string { "Cannot retrieve device capacity data (%d)." } 11004E:string { "Cannot take the device lock (%s)." } - 11005I:string { "Mounting the volume." } + 11005I:string { "Mounting the volume from %s." } 11006E:string { "Cannot read volume: failed to load the tape." } 11007D:string { "Tape is loaded." } 11008D:string { "Reading partition labels." } diff --git a/messages/make_message_src.sh b/messages/make_message_src.sh index 82488e81..13ca3861 100755 --- a/messages/make_message_src.sh +++ b/messages/make_message_src.sh @@ -78,18 +78,20 @@ make_obj() { NetBSD) # generate libtool archive for later linking mv lib${BASENAME}.a ../../lib${BASENAME}_dat.a - OBJFILE=${BASENAME}_dat.o - LTFILE=${BASENAME}_dat.lo mkdir -p .libs ../../.libs - mv ${OBJFILE} .libs - LTVERS=`libtool --version | - sed -e 's/^\([^ ]*\) (GNU \(.*\)) \(.*\)$$/\1 - GNU \2 \3/' -e q` - echo "# ${OBJFILE} - a libtool object file" > ${LTFILE} - echo "# Generated by ${LTVERS}" >> ${LTFILE} - echo "pic_object='.libs/${OBJFILE}'" >> ${LTFILE} - echo "non_pic_object=none" >> ${LTFILE} - libtool --mode=link --tag=CC cc -o ../../lib${BASENAME}_dat.la ${LTFILE} - cp ../../lib${BASENAME}_dat.a ../../.libs + LTFILES="" + LTVERS=`libtool --version | + sed -e 's/^\([^ ]*\) (GNU \(.*\)) \(.*\)$$/\1 - GNU \2 \3/' -e q` + for OBJFILE in *.o ; do + LTFILE=${OBJFILE%.o}.lo + mv ${OBJFILE} .libs + echo "# ${OBJFILE} - a libtool object file" > ${LTFILE} + echo "# Generated by ${LTVERS}" >> ${LTFILE} + echo "pic_object='.libs/${OBJFILE}'" >> ${LTFILE} + echo "non_pic_object=none" >> ${LTFILE} + LTFILES="${LTFILES} ${LTFILE}" + done + libtool --mode=link --tag=CC cc -o ../../lib${BASENAME}_dat.la ${LTFILES} ;; *) mv ${BASENAME}_dat.o ../../lib${BASENAME}_dat.a diff --git a/src/iosched/unified.c b/src/iosched/unified.c index f492e12f..14b274cb 100644 --- a/src/iosched/unified.c +++ b/src/iosched/unified.c @@ -2291,7 +2291,8 @@ int _unified_write_index_after_perm(int write_ret, struct unified_data *priv) return ret; } - ret = ltfs_write_index(ltfs_ip_id(priv->vol), SYNC_WRITE_PERM, priv->vol); + ltfs_set_commit_message_reason(SYNC_WRITE_PERM, priv->vol); + ret = ltfs_write_index(ltfs_ip_id(priv->vol), SYNC_WRITE_PERM, LTFS_FULL_INDEX, priv->vol); return ret; } diff --git a/src/libltfs/Makefile.am b/src/libltfs/Makefile.am index b3ae064d..9b9ab438 100644 --- a/src/libltfs/Makefile.am +++ b/src/libltfs/Makefile.am @@ -62,6 +62,7 @@ libltfs_la_SOURCES = \ config_file.c \ plugin.c \ periodic_sync.c \ + inc_journal.c \ arch/uuid_internal.c \ arch/filename_handling.c \ arch/time_internal.c \ diff --git a/src/libltfs/fs.c b/src/libltfs/fs.c index 790438d5..2404adce 100644 --- a/src/libltfs/fs.c +++ b/src/libltfs/fs.c @@ -657,6 +657,99 @@ void fs_split_path(char *path, char **filename, size_t len) } } +int fs_path_clean(const char *path, struct ltfs_index *idx) +{ + int ret = 0; + struct dentry *d = NULL, *parent = NULL; + char *tmp_path, *start, *end; + + CHECK_ARG_NULL(path, -LTFS_NULL_ARG); + CHECK_ARG_NULL(idx, -LTFS_NULL_ARG); + + tmp_path = strdup(path); + if (! tmp_path) { + ltfsmsg(LTFS_ERR, 10001E, "fs_path_clean: tmp_path"); + return -LTFS_NO_MEMORY; + } + + /* Get a reference count on the root dentry. Either it will be returned immediately, or it + * will be disposed later after the first path lookup. */ + acquirewrite_mrsw(&idx->root->meta_lock); + ++idx->root->numhandles; + releasewrite_mrsw(&idx->root->meta_lock); + + if (idx->root->dirty) + idx->root->dirty = false; + + /* Did the caller ask for the root dentry? */ + if (*path == '\0' || ! strcmp(path, "/")) { + goto out; + } + + start = tmp_path + 1; + end = tmp_path; + d = idx->root; + + while (end) { + end = strstr(start, "/"); + if (end) + *end = '\0'; + + acquireread_mrsw(&d->contents_lock); + + if (parent) + releaseread_mrsw(&parent->contents_lock); + parent = d; + d = NULL; + + ret = fs_directory_lookup(parent, start, &d); + if (ret < 0 || ! d) { + releaseread_mrsw(&parent->contents_lock); + fs_release_dentry(parent); + + if (ret == 0) + ret = -LTFS_NO_DENTRY; + goto out; + } + + /* Release the parent if we aren't keeping any locks on it. + * Since we know 'parent' has a child (d), it's guaranteed that parent is still linked + * into the file system tree. Therefore, fs_release_dentry is just a fancy way of + * decrementing the handle count... so do that. */ + acquirewrite_mrsw(&parent->meta_lock); + --parent->numhandles; + releasewrite_mrsw(&parent->meta_lock); + + if (d->dirty) + d->dirty = false; + + if (end) + start = end + 1; + } + + releaseread_mrsw(&parent->contents_lock); + +out: + free(tmp_path); + + return ret; +} + +int fs_dir_clean(struct dentry *d) +{ + struct name_list *list_ptr = NULL, *list_tmp = NULL; + CHECK_ARG_NULL(d, -LTFS_NULL_ARG); + + if (d->isdir) { + HASH_ITER(hh, d->child_list, list_ptr, list_tmp) { + fs_dir_clean(list_ptr->d); + } + } else + d->dirty = false; + + return 0; +} + /** * Dispose a dentry and all resources used by it, including the struct dentry itself. * @param dentry dentry to dispose. diff --git a/src/libltfs/fs.h b/src/libltfs/fs.h index 7de98153..66b8e5b5 100644 --- a/src/libltfs/fs.h +++ b/src/libltfs/fs.h @@ -146,4 +146,26 @@ int fs_path_lookup(const char *path, int flags, struct dentry **dentry, struct l */ void fs_split_path(char *path, char **filename, size_t len); +/** + * Cleanup d->dirty flag in the provided path + * + * @param path Path to search for, in UTF-8 NFC. The path should be checked + * for invalid characters by the caller. This function validates the length of each + * path component. If path points to an empty string, this function returns the + * root dentry. + * @param idx LTFS index to search. + * @return 0 on success (dentry found), -LTFS_NO_DENTRY if no dentry was found, -LTFS_NAMETOOLONG + * if any component of the path is too long, or another negative value if an internal + * (unexpected) error occurs. + */ +int fs_path_clean(const char *path, struct ltfs_index *idx); + +/** + * Cleanup d->dirty flag under the provided directory (dentry) recursively + * + * @param d dentry structure to clean. Expect a directory. + * @return 0 on success, otherwise on error + */ +int fs_dir_clean(struct dentry *d); + #endif /* __fs_helper_h */ diff --git a/src/libltfs/inc_journal.c b/src/libltfs/inc_journal.c new file mode 100644 index 00000000..9c4386e0 --- /dev/null +++ b/src/libltfs/inc_journal.c @@ -0,0 +1,707 @@ +/* +** +** OO_Copyright_BEGIN +** +** +** Copyright 2010, 2024 IBM Corp. All rights reserved. +** +** Redistribution and use in source and binary forms, with or without +** modification, are permitted provided that the following conditions +** are met: +** 1. Redistributions of source code must retain the above copyright +** notice, this list of conditions and the following disclaimer. +** 2. Redistributions in binary form must reproduce the above copyright +** notice, this list of conditions and the following disclaimer in the +** documentation and/or other materials provided with the distribution. +** 3. Neither the name of the copyright holder nor the names of its +** contributors may be used to endorse or promote products derived from +** this software without specific prior written permission. +** +** THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +** AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +** IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +** ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +** LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +** CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +** SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +** INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +** CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +** ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +** POSSIBILITY OF SUCH DAMAGE. +** +** +** OO_Copyright_END +** +************************************************************************************* +** +** COMPONENT NAME: IBM Linear Tape File System +** +** FILE NAME: inc_journal.c +** +** DESCRIPTION: Journal handling for incremental index +** +** AUTHORS: Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** +** ORIGINAL LOGIC: David Pease +** pease@coati.com +** +************************************************************************************* +*/ + +#include "ltfs.h" +#include "fs.h" +#include "inc_journal.h" + +static int _allocate_jentry(struct jentry **e, char *path, struct dentry* d) +{ + struct jentry *ent = NULL; + + *e = NULL; + + ent = calloc(1, sizeof(struct jentry)); + if (!ent) { + ltfsmsg(LTFS_ERR, 10001E, "allocating a jentry"); + return -LTFS_NO_MEMORY; + } + + ent->id.full_path = path; + ent->id.uid = d->uid; + + *e = ent; + + return 0; +} + +static inline int _dispose_jentry(struct jentry *ent) +{ + if (ent) { + if (ent->id.full_path) + free(ent->id.full_path); + free(ent); + } + + return 0; +} + +/** + * Handle created object in the tree. + * + * Caller need to grab vol->index->dirty_lock outside of this function. + * + * @param ppath parent path name of the object + * @param d dentry created + * @param vol pointer to the LTFS volume + */ +int incj_create(char *ppath, struct dentry *d, struct ltfs_volume *vol) +{ + int ret = -1, len = -1; + char *full_path = NULL; + struct jentry *ent = NULL; + struct jcreated_entry *jdir = NULL, *jd = NULL; + + /* Skip journal modification because of an error */ + if (vol->journal_err) { + return 0; + } + + /* Skip if an ancestor is already created in this session */ + TAILQ_FOREACH(jd, &vol->created_dirs, list) { + char* cp = jd->path; + if (strstr(ppath, cp) == ppath) { + return 0; + } + } + + /* Create full path of created object and jentry */ + len = asprintf(&full_path, "%s/%s", ppath, d->name.name); + if (len < 0) { + ltfsmsg(LTFS_ERR, 10001E, "full path of a jentry"); + vol->journal_err = true; + return -LTFS_NO_MEMORY; + } + + ret = _allocate_jentry(&ent, full_path, d); + if (ret < 0) { + vol->journal_err = true; + free(full_path); + return ret; + } + + ent->reason = CREATE; + ent->dentry = d; + + HASH_ADD(hh, vol->journal, id, sizeof(struct jentry), ent); + + if (d->isdir) { + jdir = calloc(1, sizeof(struct jcreated_entry)); + if (!jdir) { + ltfsmsg(LTFS_ERR, 10001E, "allocating a jcreated_entry"); + return -LTFS_NO_MEMORY; + } + + /* NOTE: Use same pointer of ent because it's life is same */ + jdir->path = ent->id.full_path; + + TAILQ_INSERT_TAIL(&vol->created_dirs, jdir, list); + } + + return 0; +} + +/** + * Handle modified file in the tree. + * + * Caller need to grab vol->index->dirty_lock outside of this function. + * + * @param path path name of the object + * @param d dentry to be modified (for recording uid) + * @param vol pointer to the LTFS volume + */ +int incj_modify(char *path, struct dentry *d, struct ltfs_volume *vol) +{ + int ret = -1; + struct jentry *ent = NULL; + struct jcreated_entry *jd = NULL; + + /* Skip journal modification because of an error */ + if (vol->journal_err) { + return 0; + } + + /* Skip journal modification because it is already existed */ + HASH_FIND(hh, vol->journal, &d->uid, sizeof(struct jentry), ent); + if (ent) { + return 0; + } + + /* Skip if an ancestor is already created in this session */ + TAILQ_FOREACH(jd, &vol->created_dirs, list) { + char *cp = jd->path; + if (strstr(path, cp) == path) { + return 0; + } + } + + ret = _allocate_jentry(&ent, path, d); + if (ret < 0) { + vol->journal_err = true; + return ret; + } + + ent->reason = MODIFY; + ent->dentry = d; + + HASH_ADD(hh, vol->journal, id, sizeof(struct jentry), ent); + + return 0; +} + +/** + * Handle deleted file in the tree. + * + * Caller need to grab vol->index->dirty_lock outside of this function. + * + * @param path path name of the object + * @param d dentry to be removed (for recording uid) + * @param vol pointer to the LTFS volume + */ +int incj_rmfile(char *path, struct dentry *d, struct ltfs_volume *vol) +{ + int ret = -1; + char *full_path = NULL; + struct journal_id id; + struct jentry *ent = NULL; + struct jcreated_entry *jd = NULL; + + /* Skip journal modification because of an error */ + if (vol->journal_err) { + return 0; + } + + id.full_path = path; + id.uid = d->uid; + HASH_FIND(hh, vol->journal, &id, sizeof(struct jentry), ent); + if (ent) { + if (ent->reason == CREATE) { + /* + * Remove the entry because this file is newly created and deleted + * in one incremental index session + */ + HASH_DEL(vol->journal, ent); + return 0; + } else if (ent->reason == MODIFY) { + /* + * Override the existing entry to DELETE_FILE record. + */ + ent->reason = DELETE_FILE; + ent->dentry = NULL; + return 0; + } + } + + /* Skip if an ancestor is already created in this session */ + TAILQ_FOREACH(jd, &vol->created_dirs, list) { + char *cp = jd->path; + if (strstr(path, cp) == path) { + return 0; + } + } + + /* Create full path of deleted object and jentry */ + full_path = strdup(path); + if (!full_path) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a path for deleted file"); + vol->journal_err = true; + return -LTFS_NO_MEMORY; + } + + ret = _allocate_jentry(&ent, full_path, d); + if (ret < 0) { + vol->journal_err = true; + return ret; + } + + ent->reason = DELETE_FILE; + ent->name.percent_encode = d->name.percent_encode; + ent->name.name = strdup(d->name.name); + if (!ent->name.name) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a name of deleted file"); + vol->journal_err = true; + return -LTFS_NO_MEMORY; + } + + HASH_ADD(hh, vol->journal, id, sizeof(struct jentry), ent); + + return 0; +} + +/** + * Handle deleted directory in the tree. + * + * Caller need to grab vol->index->dirty_lock outside of this function. + * + * @param path path name of the object + * @param d dentry to be removed (for recording uid) + * @param vol pointer to the LTFS volume + */ +int incj_rmdir(char *path, struct dentry *d, struct ltfs_volume *vol) +{ + int ret = -1; + char *full_path = NULL; + struct jentry *ent = NULL, *je = NULL, *tmp = NULL; + struct jcreated_entry *jd = NULL, *dtmp = NULL; + + /* Skip journal modification because of an error */ + if (vol->journal_err) { + return 0; + } + + /* + * 1. Remove entry from created_dirs if created directory is removed in a same session + * 2. Skip if an ancestor is already created in this session + */ + TAILQ_FOREACH_SAFE(jd, &vol->created_dirs, list, dtmp) { + char *cp = jd->path; + if (strstr(path, cp) == path) { + if (!strcmp(path, cp)) { + TAILQ_REMOVE(&vol->created_dirs, jd, list); + /* + * NOTE: + * Do not free jd->path because it shall be freed into _dispose_jentry. + * jentry::id.full_path and jd->path points the same address + */ + } else { + return 0; + } + } + } + + /* Need to find existing children under this directory */ + HASH_ITER(hh, vol->journal, je, tmp) { + if (strstr(je->id.full_path, path) == je->id.full_path) { + HASH_DEL(vol->journal, je); + _dispose_jentry(je); + } + } + + /* Create full path of created object and jentry */ + full_path = strdup(path); + if (!full_path) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a path of deleted directory"); + vol->journal_err = true; + return -LTFS_NO_MEMORY; + } + + ret = _allocate_jentry(&ent, full_path, d); + if (ret < 0) { + vol->journal_err = true; + return ret; + } + + ent->reason = DELETE_DIRECTORY; + ent->name.percent_encode = d->name.percent_encode; + ent->name.name = strdup(d->name.name); + if (!ent->name.name) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a name of deleted directory"); + vol->journal_err = true; + return -LTFS_NO_MEMORY; + } + + HASH_ADD(hh, vol->journal, id, sizeof(struct jentry), ent); + + return 0; +} + +int incj_dispose_jentry(struct jentry *ent) +{ + return (_dispose_jentry(ent)); +} + +/** + * Clear all entries into the incremental journal + */ +int incj_clear(struct ltfs_volume *vol) +{ + struct jentry *je = NULL, *tmp = NULL; + struct jcreated_entry *jd = NULL, *dtmp = NULL; + + TAILQ_FOREACH_SAFE(jd, &vol->created_dirs, list, dtmp) { + TAILQ_REMOVE(&vol->created_dirs, jd, list); + } + + HASH_ITER(hh, vol->journal, je, tmp) { + HASH_DEL(vol->journal, je); + _dispose_jentry(je); + } + + return 0; +} + +/** + * Sort function for dump incremental journal + */ +static int _by_path(const struct jentry *a, const struct jentry *b) +{ + int ret = 0; + + ret = strcmp(a->id.full_path, b->id.full_path); + if (!ret) { + if (a->id.uid > b->id.uid) + ret = 1; + else + ret = -1; + } + + return ret; +} + +static inline int dig_path(char *p, struct ltfs_index *idx) +{ + int ret = 0; + char *path; + + path = strdup(p); + if (! path) { + ltfsmsg(LTFS_ERR, 10001E, "dig_path: path"); + return -LTFS_NO_MEMORY; + } + + ret = fs_path_clean(path, idx); + + free(path); + + return ret; +} + +void incj_sort(struct ltfs_volume *vol) +{ + HASH_SORT(vol->journal, _by_path); +} + +/** + * This is a function for debug. Print contents of the journal and the created + * directory list to stdout. + */ +void incj_dump(struct ltfs_volume *vol) +{ + char *prev_parent = NULL, *parent, *filename; + struct jcreated_entry *jd = NULL, *dtmp = NULL; + struct jentry *ent = NULL, *tmp = NULL; + char *reason[] = { "CREATE", "MODIFY", "DELFILE", "DELDIR" }; + + printf("===============================================================================\n"); + TAILQ_FOREACH_SAFE(jd, &vol->created_dirs, list, dtmp) { + printf("CREATED_DIR: %s\n", jd->path); + TAILQ_REMOVE(&vol->created_dirs, jd, list); + } + + printf("--------------------------------------------------------------------------------\n"); + incj_sort(vol); + HASH_ITER(hh, vol->journal, ent, tmp) { + printf("JOURNAL: %s, %llu, %s, ", ent->id.full_path, (unsigned long long)ent->id.uid, reason[ent->reason]); + if (!ent->dentry) + printf("no-dentry\n"); + else { + if (ent->dentry->isdir) { + printf("dir\n"); + if (ent->reason == CREATE) + fs_dir_clean(ent->dentry); + } else + printf("file\n"); + + parent = strdup(ent->id.full_path); + fs_split_path(parent, &filename, strlen(parent) + 1); + + if (prev_parent) { + if (strcmp(prev_parent, parent)) { + dig_path(parent, vol->index); + } + free(prev_parent); + } else { + dig_path(parent, vol->index); + } + prev_parent = parent; + ent->dentry->dirty = false; + } + + HASH_DEL(vol->journal, ent); + _dispose_jentry(ent); + } + + if (prev_parent) free(prev_parent); + + return; +} + +int incj_create_path_helper(const char *dpath, struct incj_path_helper **pm, struct ltfs_volume *vol) +{ + struct incj_path_helper *ipm; + char *wp = NULL, *tmp = NULL, *dname = NULL; + int ret = 0; + + *pm = NULL; + + ipm = calloc(1, sizeof(struct incj_path_helper)); + if (!ipm) { + ltfsmsg(LTFS_ERR, 10001E, "allocating a path helper"); + return -LTFS_NO_MEMORY; + } + + if (dpath[0] != '/') { + /* Provided path must be a absolute path */ + ltfsmsg(LTFS_ERR, 17302E, dpath); + free(ipm); + return -LTFS_INVALID_PATH; + } + + ipm->vol = vol; + + if (strcmp(dpath, "/") == 0) { + /* Provided path is the root, return good */ + *pm = ipm; + return 0; + } + + wp = strdup(dpath); + if (!wp) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a directory path for path helper"); + free(ipm); + return -LTFS_NO_MEMORY; + } + + for (dname = strtok_r(wp, "/", &tmp); dname != NULL; dname = strtok_r(NULL, "/", &tmp)) { + ret = incj_push_directory(dname, ipm); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 17305E, ret); + free(wp); + incj_destroy_path_helper(ipm); + return ret; + } + } + + free(wp); + *pm = ipm; + + return 0; +} + +int incj_destroy_path_helper(struct incj_path_helper *pm) +{ + struct incj_path_element *cur, *next; + + cur = pm->head; + + while (cur) { + next = cur->next; + if (cur->d) + fs_release_dentry(cur->d); + if (cur->name) + free(cur->name); + free(cur); + cur = next; + } + + free(pm); + return 0; +} + +int incj_push_directory(char *name, struct incj_path_helper *pm) +{ + int ret = 0; + struct incj_path_element *ipelm = NULL, *cur_tail = NULL; + struct dentry *parent = NULL; + + ipelm = calloc(1, sizeof(struct incj_path_element)); + if (!ipelm) { + ltfsmsg(LTFS_ERR, 10001E, "allocating a path element on push"); + return -LTFS_NO_MEMORY; + } + + /* Set name field of new path element */ + ipelm->name = strdup(name); + if (!ipelm->name) { + ltfsmsg(LTFS_ERR, 10001E, "duplicating a path of pushing directory"); + incj_destroy_path_helper(pm); + return -LTFS_NO_MEMORY; + } + + /* Set dentry field of new path element */ + if (pm->elems) + parent = pm->tail->d; + else + parent = pm->vol->index->root; + + ret = fs_directory_lookup(parent, name, &ipelm->d); + if (ret) { + ltfsmsg(LTFS_ERR, 17306E, ret); + free(ipelm->name); + free(ipelm); + incj_destroy_path_helper(pm); + return -LTFS_INVALID_PATH; + } + + /* Modify path chain and # of elements */ + if (!pm->elems) { + pm->head = ipelm; + pm->tail = ipelm; + } else { + cur_tail = pm->tail; + cur_tail->next = ipelm; + ipelm->prev = cur_tail; + pm->tail = ipelm; + } + + pm->elems++; + + return 0; +} + +int incj_pop_directory(struct incj_path_helper *pm) +{ + struct incj_path_element *cur_tail = NULL, *new_tail = NULL; + + if (!pm->elems) { + /* Must have one or more elements */ + return -LTFS_UNEXPECTED_VALUE; + } + + cur_tail = pm->tail; + new_tail = cur_tail->prev; + + new_tail->next = NULL; + pm->tail = new_tail; + + pm->elems--; + if (!pm->elems) { + pm->head = NULL; + } + + if (cur_tail->d) + fs_release_dentry(cur_tail->d); + if (cur_tail->name) + free(cur_tail->name); + free(cur_tail); + + return 0; +} + +int incj_compare_path(struct incj_path_helper *now, struct incj_path_helper *next, + int *matches, int *pops, bool *perfect_match) +{ + int ret = 0, matched = 0; + struct incj_path_element *cur1 = NULL, *cur2 = NULL; + + *matches = 0; + *pops = 0; + *perfect_match = false; + + cur1 = now->head; + cur2 = next->head; + + if (!cur1 && !cur2) { + /* Both are root */ + *perfect_match = true; + return 0; + } + + while (cur1 && cur2) { + if (cur1->d != cur2->d) + break; + matched++; + cur1 = cur1->next; + cur2 = cur2->next; + } + + *matches = matched; + *pops = now->elems - *matches; + + if (!cur1 && !cur2) + *perfect_match = true; + + return ret; +} + +char* incj_get_path(struct incj_path_helper *pm) +{ + char *path = NULL, *path_old = NULL; + struct incj_path_element *cur = NULL; + int ret = 0; + + cur = pm->head; + + if (!cur) { + /* Root directory */ + ret = asprintf(&path, "/"); + if (ret < 0) { + /* memory allocation error */ + return NULL; + } + return path; + } + + while (cur) { + if (path) path_old = path; + ret = asprintf(&path, "%s/%s", path_old, cur->name); + if (ret < 0) { + /* memory allocation error */ + free(path_old); + return NULL; + } + free(path_old); + + cur++; + } + + if (path) path_old = path; + ret = asprintf(&path, "/%s", path_old); + if (ret < 0) { + /* memory allocation error */ + free(path_old); + return NULL; + } + free(path_old); + + return path; +} diff --git a/src/libltfs/inc_journal.h b/src/libltfs/inc_journal.h new file mode 100644 index 00000000..14784857 --- /dev/null +++ b/src/libltfs/inc_journal.h @@ -0,0 +1,137 @@ +/* +** +** OO_Copyright_BEGIN +** +** +** Copyright 2010, 2024 IBM Corp. All rights reserved. +** +** Redistribution and use in source and binary forms, with or without +** modification, are permitted provided that the following conditions +** are met: +** 1. Redistributions of source code must retain the above copyright +** notice, this list of conditions and the following disclaimer. +** 2. Redistributions in binary form must reproduce the above copyright +** notice, this list of conditions and the following disclaimer in the +** documentation and/or other materials provided with the distribution. +** 3. Neither the name of the copyright holder nor the names of its +** contributors may be used to endorse or promote products derived from +** this software without specific prior written permission. +** +** THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +** AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +** IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +** ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +** LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +** CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +** SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +** INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +** CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +** ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +** POSSIBILITY OF SUCH DAMAGE. +** +** +** OO_Copyright_END +** +************************************************************************************* +** +** COMPONENT NAME: IBM Linear Tape File System +** +** FILE NAME: inc_journal.h +** +** DESCRIPTION: Journal handling for incremental index +** +** AUTHORS: Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** +** ORIGINAL LOGIC: David Pease +** pease@coati.com +** +************************************************************************************* +*/ + +#include "queue.h" + +#ifndef __inc_journal_h__ +#define __inc_journal_h__ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * Enumeration of reasons for an entry + */ +enum journal_reason { + CREATE = 0, /**< Newly created, need to be 0 for debug */ + MODIFY, /**< Modified */ + DELETE_FILE, /**< File is deleted */ + DELETE_DIRECTORY, /**< Directory is deleted */ +}; + +/** + * Identifier of journal entry for handling multiple changes in one session + */ +struct journal_id { + char *full_path; /**< Full path name of the target */ + uint64_t uid; /**< i-node number of the target */ +}; + +/** + * Journal entry + */ +struct jentry { + struct journal_id id; /**< ID of the journal entry (key of the hash table) */ + enum journal_reason reason; /**< Reason of the entry */ + struct dentry *dentry; /**< Target dentry if required */ + struct ltfs_name name; /**< Name of entry for delete */ + UT_hash_handle hh; +}; + +/** + * Created directory list + */ +struct jcreated_entry { + TAILQ_ENTRY(jcreated_entry) list; /**< Pointers for linked list of requests */ + char *path; +}; + +/** + * + */ +struct incj_path_element { + struct incj_path_element *prev; + struct incj_path_element *next; + char* name; + struct dentry *d; +}; + +struct incj_path_helper { + struct incj_path_element *head; + struct incj_path_element *tail; + struct ltfs_volume *vol; + unsigned int elems; +}; + +int incj_create(char *ppath, struct dentry *d, struct ltfs_volume *vol); +int incj_modify(char *path, struct dentry *d, struct ltfs_volume *vol); +int incj_rmfile(char *path, struct dentry *d, struct ltfs_volume *vol); +int incj_rmdir(char *path, struct dentry *d, struct ltfs_volume *vol); +int incj_dispose_jentry(struct jentry *ent); +int incj_clear(struct ltfs_volume *vol); +void incj_sort(struct ltfs_volume *vol); +void incj_dump(struct ltfs_volume *vol); + +int incj_create_path_helper(const char *path, struct incj_path_helper **pm, struct ltfs_volume *vol); +int incj_destroy_path_helper(struct incj_path_helper *pm); +int incj_push_directory(char *name, struct incj_path_helper *pm); +int incj_pop_directory(struct incj_path_helper *pm); +int incj_compare_path(struct incj_path_helper *p1, struct incj_path_helper *p2, + int *matches, int *pops, bool *perfect_match); +char* incj_get_path(struct incj_path_helper *pm); + +#ifdef __cplusplus +} +#endif + +#endif /* __inc_journal_h__ */ diff --git a/src/libltfs/ltfs.c b/src/libltfs/ltfs.c index c0c624b3..c11b4b53 100644 --- a/src/libltfs/ltfs.c +++ b/src/libltfs/ltfs.c @@ -70,6 +70,7 @@ #include "xattr.h" #include "xml_libltfs.h" #include "label.h" +#include "inc_journal.h" #include "arch/version.h" #include "arch/filename_handling.h" #include "libltfs/arch/errormap.h" @@ -428,6 +429,9 @@ int ltfs_volume_alloc(const char *execname, struct ltfs_volume **volume) } } + newvol->journal = NULL; + TAILQ_INIT(&newvol->created_dirs); + *volume = newvol; return 0; @@ -470,8 +474,10 @@ void _ltfs_volume_free(bool force, struct ltfs_volume **volume) free((*volume)->mountpoint); if ((*volume)->t_attr) free((*volume)->t_attr); - if ((*volume)->index_cache_path) - free((*volume)->index_cache_path); + if ((*volume)->index_cache_path_w) + free((*volume)->index_cache_path_w); + if ((*volume)->index_cache_path_r) + free((*volume)->index_cache_path_r); destroy_mrsw(&(*volume)->lock); ltfs_thread_mutex_destroy(&(*volume)->reval_lock); ltfs_thread_cond_destroy(&(*volume)->reval_cond); @@ -601,6 +607,13 @@ int ltfs_test_unit_ready(struct ltfs_volume *vol) ret = ltfs_get_volume_lock(false, vol); if (ret < 0) return ret; + + if (vol->mount_type == MOUNT_ROLLBACK_META) { + /* Return good when volume is meta-data mount mode */ + releaseread_mrsw(&vol->lock); + return 0; + } + ret = tape_device_lock(vol->device); if (ret == -LTFS_DEVICE_FENCED) { ret = ltfs_wait_revalidation(vol); @@ -697,6 +710,13 @@ int ltfs_capacity_data(struct device_capacity *cap, struct ltfs_volume *vol) ret = ltfs_get_volume_lock(false, vol); if (ret < 0) return ret; + + if (vol->mount_type == MOUNT_ROLLBACK_META) { + /* Return good when volume is meta-data mount mode */ + releaseread_mrsw(&vol->lock); + return 0; + } + ret = ltfs_capacity_data_unlocked(cap, vol); if (ret == -LTFS_DEVICE_FENCED) { ret = ltfs_wait_revalidation(vol); @@ -1581,7 +1601,7 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec char *vl_print = NULL; char *barcode = NULL; - ltfsmsg(LTFS_INFO, 11005I); + ltfsmsg(LTFS_INFO, 11005I, "device"); CHECK_ARG_NULL(vol, -LTFS_NULL_ARG); @@ -1780,7 +1800,7 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec goto out_unlock; } else { INTERRUPTED_GOTO(ret, out_unlock); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, false, vol); if (ret < 0) { if (read_ip) ltfsmsg(LTFS_ERR, 11024E); /* read IP Index failed */ @@ -1813,14 +1833,15 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec goto out_unlock; } else { INTERRUPTED_GOTO(ret, out_unlock); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, false, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 11021E); /* read DP Index failed */ goto out_unlock; } INTERRUPTED_GOTO(ret, out_unlock); ltfsmsg(LTFS_INFO, 11022I); - ret = ltfs_write_index(vol->label->partid_ip, SYNC_RECOVERY, vol); + ltfs_set_commit_message_reason(SYNC_RECOVERY, vol); + ret = ltfs_write_index(vol->label->partid_ip, SYNC_RECOVERY, LTFS_FULL_INDEX, vol); if (ret < 0) goto out_unlock; } @@ -1842,7 +1863,7 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec goto out_unlock; } else { INTERRUPTED_GOTO(ret, out_unlock); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, false, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 11024E); /* read IP Index failed */ goto out_unlock; @@ -1869,17 +1890,17 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec INTERRUPTED_GOTO(ret, out_unlock); if(gen != 0 && gen != vol->index->generation) { if(is_worm_recovery_mount){ - ret = ltfs_traverse_index_no_eod(vol, ltfs_ip_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_no_eod(vol, ltfs_ip_id(vol), gen, false, NULL, NULL, NULL); if(ret < 0) - ret = ltfs_traverse_index_no_eod(vol, ltfs_dp_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_no_eod(vol, ltfs_dp_id(vol), gen, false, NULL, NULL, NULL); } else if(vol->traverse_mode == TRAVERSE_FORWARD){ - ret = ltfs_traverse_index_forward(vol, ltfs_ip_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_forward(vol, ltfs_ip_id(vol), gen, false, NULL, NULL, NULL); if(ret < 0) - ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), gen, false, NULL, NULL, NULL); } else { - ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), gen, false, NULL, NULL, NULL); if(ret < 0) - ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), gen, NULL, NULL, NULL); + ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), gen, false, NULL, NULL, NULL); } if (ret < 0) { ltfsmsg(LTFS_ERR, 17079E, gen); @@ -1905,13 +1926,6 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec if (vol->index->uid_number == 0) ltfsmsg(LTFS_WARN, 11307W, vol->label->vol_uuid); - /* Clear the commit message so it doesn't carry over from the previous session */ - /* TODO: is this the right place to clear the commit message? */ - if (vol->index->commit_message) { - free(vol->index->commit_message); - vol->index->commit_message = NULL; - } - /* If we reach this point, both partitions end in an index file. */ vol->ip_index_file_end = true; vol->dp_index_file_end = true; @@ -1960,6 +1974,62 @@ int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool rec return ret; } +/** + * Mount LTFS volume from provided index file + * Volume shall be mounted as read-only to avoid illegal data update. + * + * @param filename file name of the index + * @param label_check check label on tape (for R/O mount with data access) on true + * @param vol the volume to load + * @return 0 on success or a negative value on error. + */ +int ltfs_mount_indexfile(char* filename, bool label_check, struct ltfs_volume *vol) +{ + int ret = 0; + + ltfsmsg(LTFS_INFO, 11005I, filename); + + CHECK_ARG_NULL(filename, -LTFS_NULL_ARG); + CHECK_ARG_NULL(vol, -LTFS_NULL_ARG); + + if (label_check) { + /* load tape, read indexes, set compression */ + ret = ltfs_start_mount(false, vol); + if (ret < 0) { + /* ltfs_start_mount() generated an appropriate error message */ + goto out_unlock; + } + ltfsmsg(LTFS_DEBUG, 11013D); /* partition labels are valid */ + vol->mount_type = MOUNT_ROLLBACK; + } else { + /* Assume 512KB block*/ + vol->label->blocksize = 512 * KB; + vol->mount_type = MOUNT_ROLLBACK_META; + } + + vol->first_locate.tv_sec = 0; + vol->first_locate.tv_nsec = 0; + + ret = ltfs_read_indexfile(filename, false, vol); + + if (label_check) { + if (strcmp(vol->index->vol_uuid, vol->label->vol_uuid)) { + /* + * Volume UUID in label and it on index is not matched. + * Actual UUID is not printed to avoid illegal modification by hand. + */ + ltfsmsg(LTFS_ERR, 17293E); + ret = -LTFS_LABEL_MISMATCH; + } + } + +out_unlock: + if (ret < 0 && vol->index) + ltfs_index_free(&vol->index); + + return ret; +} + /** * Load cartridge attribute varues from CM * @param vol the volume to get attribute @@ -2057,13 +2127,14 @@ int ltfs_unmount(char *reason, struct ltfs_volume *vol) start: ret = ltfs_get_volume_lock(true, vol); - if (!ret) { + if (!ret && vol->mount_type != MOUNT_ROLLBACK_META) { + ret = tape_get_cart_volume_lock_status(vol->device, &vollock); if (vol->mount_type == MOUNT_NORMAL && (ltfs_is_dirty(vol) || vol->index->selfptr.partition != ltfs_ip_id(vol)) && (vollock != PWE_MAM_IP && vollock != PWE_MAM_BOTH)) { - ret = ltfs_write_index(ltfs_ip_id(vol), reason, vol); + ret = ltfs_write_index(ltfs_ip_id(vol), reason, LTFS_FULL_INDEX, vol); if (NEED_REVAL(ret)) { ret = ltfs_revalidate(true, vol); if (ret == 0) { @@ -2429,10 +2500,10 @@ size_t ltfs_max_cache_size(struct ltfs_volume *vol) * @param vol LTFS volume * @return 0 on success or a negative value on error */ -int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) +int ltfs_write_index(char partition, char *reason, enum ltfs_index_type type, struct ltfs_volume *vol) { int ret, ret_mam; - struct tape_offset old_selfptr, old_backptr; + struct tape_offset old_selfptr, old_backptr, old_selfptr_inc, old_backptr_inc; struct ltfs_timespec modtime_old = { .tv_sec = 0, .tv_nsec = 0 }; bool generation_inc = false; struct tc_position physical_selfptr, current_position; @@ -2485,12 +2556,15 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) (vol->ip_index_file_end && vol->index->selfptr.partition == ltfs_ip_id(vol)))) { /* Surpress on-disk index cache write on the recursive call */ - cache_path_save = vol->index_cache_path; - vol->index_cache_path = NULL; - ret = ltfs_write_index(ltfs_dp_id(vol), reason, vol); + cache_path_save = vol->index_cache_path_w; + vol->index_cache_path_w = NULL; + + type = LTFS_FULL_INDEX; + incj_clear(vol); /* Clear incremental journal data */ + ret = ltfs_write_index(ltfs_dp_id(vol), reason, type, vol); /* Restore cache path to handle on-disk index cache */ - vol->index_cache_path = cache_path_save; + vol->index_cache_path_w = cache_path_save; cache_path_save = NULL; if (NEED_REVAL(ret)) @@ -2519,16 +2593,17 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) write_perm = true; reason = SYNC_WRITE_PERM; + ltfs_set_commit_message_reason(SYNC_WRITE_PERM, vol); } /* ignore return value: we want to keep trying even if, e.g., the DP fills up */ } /* update index generation */ if (ltfs_is_dirty(vol)) { - modtime_old = vol->index->mod_time; generation_inc = true; - get_current_timespec(&vol->index->mod_time); ++vol->index->generation; + modtime_old = vol->index->mod_time; + get_current_timespec(&vol->index->mod_time); } /* locate to append position */ @@ -2543,19 +2618,28 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) } /* update back pointer */ - old_backptr = vol->index->backptr; - if (vol->index->selfptr.partition == ltfs_dp_id(vol)) - memcpy(&vol->index->backptr, &vol->index->selfptr, sizeof(struct tape_offset)); + if (type == LTFS_FULL_INDEX) { + old_backptr = vol->index->backptr; + if (vol->index->selfptr.partition == ltfs_dp_id(vol)) + memcpy(&vol->index->backptr, &vol->index->selfptr, sizeof(struct tape_offset)); + } else { + old_backptr_inc = vol->index->backptr_inc; + memcpy(&vol->index->backptr_inc, &vol->index->selfptr_inc, sizeof(struct tape_offset)); + } /* update self pointer */ ret = tape_get_position(vol->device, &physical_selfptr); if (ret < 0) { ltfsmsg(LTFS_ERR, 11081E, ret); - if (generation_inc) { - vol->index->mod_time = modtime_old; - --vol->index->generation; + if (type == LTFS_FULL_INDEX) { + if (generation_inc) { + vol->index->mod_time = modtime_old; + --vol->index->generation; + } + vol->index->backptr = old_backptr; + } else { + vol->index->backptr_inc = old_backptr_inc; } - vol->index->backptr = old_backptr; goto out_write_perm; } @@ -2587,12 +2671,17 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) ret = tape_write_filemark(vol->device, 0, true, true, false); // Flush data before writing FM if (ret < 0) { ltfsmsg(LTFS_ERR, 11326E, ret); - if (generation_inc) { - vol->index->mod_time = modtime_old; - --vol->index->generation; + if (type == LTFS_FULL_INDEX) { + if (generation_inc) { + vol->index->mod_time = modtime_old; + --vol->index->generation; + } + vol->index->backptr = old_backptr; + vol->index->selfptr = old_selfptr; + } else { + vol->index->backptr_inc = old_backptr_inc; + vol->index->selfptr_inc = old_selfptr_inc; } - vol->index->backptr = old_backptr; - vol->index->selfptr = old_selfptr; if (IS_WRITE_PERM(-ret)) update_vollock = true; @@ -2607,12 +2696,18 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) ret = tape_write_filemark(vol->device, 1, true, true, true); // immediate WFM if (ret < 0) { ltfsmsg(LTFS_ERR, 11082E, ret); - if (generation_inc) { - vol->index->mod_time = modtime_old; - --vol->index->generation; + + if (type == LTFS_FULL_INDEX) { + if (generation_inc) { + vol->index->mod_time = modtime_old; + --vol->index->generation; + } + vol->index->backptr = old_backptr; + vol->index->selfptr = old_selfptr; + } else { + vol->index->backptr_inc = old_backptr_inc; + vol->index->selfptr_inc = old_selfptr_inc; } - vol->index->backptr = old_backptr; - vol->index->selfptr = old_selfptr; if (IS_WRITE_PERM(-ret)) update_vollock = true; @@ -2621,15 +2716,21 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) } /* Actually write index to tape and disk if vol->index_cache_path is existed */ - ret = xml_schema_to_tape(reason, vol); + ret = xml_schema_to_tape(reason, type, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 11083E, ret); - if (generation_inc) { - vol->index->mod_time = modtime_old; - --vol->index->generation; + + if (type == LTFS_FULL_INDEX) { + if (generation_inc) { + vol->index->mod_time = modtime_old; + --vol->index->generation; + } + vol->index->backptr = old_backptr; + vol->index->selfptr = old_selfptr; + } else { + vol->index->backptr_inc = old_backptr_inc; + vol->index->selfptr_inc = old_selfptr_inc; } - vol->index->backptr = old_backptr; - vol->index->selfptr = old_selfptr; if (IS_WRITE_PERM(-ret)) update_vollock = true; @@ -2637,15 +2738,16 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) goto out_write_perm; } - /* Update MAM parameters. */ - if (partition == ltfs_ip_id(vol)) - vol->ip_index_file_end = true; - else /* partition == ltfs_dp_id(vol) */ - vol->dp_index_file_end = true; + if (type == LTFS_FULL_INDEX) { + /* Update MAM parameters. */ + if (partition == ltfs_ip_id(vol)) + vol->ip_index_file_end = true; + else /* partition == ltfs_dp_id(vol) */ + vol->dp_index_file_end = true; - /* The MAM may be inaccessible, or it may not be available on this medium. Either way, - * ignore failures when updating MAM parameters. */ - ltfs_update_cart_coherency(vol); + /* The MAM may be inaccessible, or it may not be available on this medium. Either way, + * ignore failures when updating MAM parameters. */ + ltfs_update_cart_coherency(vol); ltfsmsg(LTFS_INFO, 17236I, bc_print, @@ -2654,20 +2756,41 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) (unsigned long long)vol->index->selfptr.block, tape_get_serialnumber(vol->device)); - /* update append position */ - if (partition == ltfs_ip_id(vol)) { - tape_set_ip_append_position(vol->device, ltfs_part_id2num(partition, vol), - vol->index->selfptr.block - 1); - } + /* update append position */ + if (partition == ltfs_ip_id(vol)) { + tape_set_ip_append_position(vol->device, ltfs_part_id2num(partition, vol), + vol->index->selfptr.block - 1); + } - if (dcache_initialized(vol)) { - dcache_set_dirty(false, vol); - if (generation_inc) { - dcache_set_generation(vol->index->generation, vol); + if (dcache_initialized(vol)) { + dcache_set_dirty(false, vol); + if (generation_inc) { + dcache_set_generation(vol->index->generation, vol); + } } - } - ltfs_unset_index_dirty(true, vol->index); + /* Clear incremental index metrix */ + vol->index->backptr_inc.partition = 0; + vol->index->backptr_inc.block = 0; + vol->index->selfptr_inc.partition = 0; + vol->index->selfptr_inc.block = 0; + + incj_clear(vol); /* Clear incremental journal data */ + ltfs_unset_index_dirty(true, vol->index); + } else { + ltfsmsg(LTFS_INFO, 17300I, + bc_print, + (unsigned long long)vol->index->generation, + vol->index->selfptr_inc.partition, + (unsigned long long)vol->index->selfptr_inc.block, + tape_get_serialnumber(vol->device)); + ltfsmsg(LTFS_INFO, 17301I, + (unsigned long long)vol->index->generation, + vol->index->backptr.partition, + (unsigned long long)vol->index->backptr.block, + vol->index->backptr_inc.partition, + (unsigned long long)vol->index->backptr_inc.block); + } out_write_perm: if (write_perm) { @@ -2701,11 +2824,11 @@ int ltfs_write_index(char partition, char *reason, struct ltfs_volume *vol) * Write down the state of the current LTFS file system to a XML file on disk. * Acquires a write lock on vol->index->root->lock. * @param work_dir LTFS work directory. - * @param need_gen include generation number to file name + * @param id partition ID or 0 * @param vol LTFS volume * @return 0 on success or a negative value on error */ -int ltfs_save_index_to_disk(const char *work_dir, char * reason, bool need_gen, struct ltfs_volume *vol) +int ltfs_save_index_to_disk(const char *work_dir, char * reason, char id, struct ltfs_volume *vol) { char *path; int ret; @@ -2717,13 +2840,13 @@ int ltfs_save_index_to_disk(const char *work_dir, char * reason, bool need_gen, /* Write the schema to a file on disk */ ltfsmsg(LTFS_DEBUG, 17182D, vol->label->vol_uuid, vol->label->barcode); - if (need_gen) { - if (strcmp(vol->label->barcode, " ")) - ret = asprintf(&path, "%s/%s-%d.schema", work_dir, vol->label->barcode, vol->index->generation); + if (id) { + if (HAVE_BARCODE(vol)) + ret = asprintf(&path, "%s/%s-%d-%c.schema", work_dir, vol->label->barcode, vol->index->generation, id); else - ret = asprintf(&path, "%s/%s-%d.schema", work_dir, vol->label->vol_uuid, vol->index->generation); + ret = asprintf(&path, "%s/%s-%d-%c.schema", work_dir, vol->label->vol_uuid, vol->index->generation, id); } else { - if (strcmp(vol->label->barcode, " ")) + if (HAVE_BARCODE(vol)) ret = asprintf(&path, "%s/%s.schema", work_dir, vol->label->barcode); else ret = asprintf(&path, "%s/%s.schema", work_dir, vol->label->vol_uuid); @@ -3140,7 +3263,8 @@ int ltfs_format_tape(struct ltfs_volume *vol, int density_code, bool destructive if (ret < 0) return ret; ltfsmsg(LTFS_INFO, 11278I, vol->label->partid_dp); /* "Writing Index to ..." */ - ret = ltfs_write_index(vol->label->partid_dp, SYNC_FORMAT, vol); + ltfs_set_commit_message_reason(SYNC_FORMAT, vol); + ret = ltfs_write_index(vol->label->partid_dp, SYNC_FORMAT, LTFS_FULL_INDEX, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 11279E, vol->label->partid_dp, ret); return ret; @@ -3153,7 +3277,7 @@ int ltfs_format_tape(struct ltfs_volume *vol, int density_code, bool destructive if (ret < 0) return ret; ltfsmsg(LTFS_INFO, 11278I, vol->label->partid_ip); /* "Writing Index to ..." */ - ret = ltfs_write_index(vol->label->partid_ip, SYNC_FORMAT, vol); + ret = ltfs_write_index(vol->label->partid_ip, SYNC_FORMAT, LTFS_FULL_INDEX, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 11279E, vol->label->partid_ip, ret); return ret; @@ -3554,13 +3678,13 @@ int ltfs_revalidate(bool have_write_lock, struct ltfs_volume *vol) /** * Write index to tape if the index is dirty, and if there is space available * on the data partition. - * @param vol LTFS volume + * @param reason the reason to write an index * @param index_locking Take index lock while writing an index * @param type index type to write * @param vol LTFS volume * @return 0 on success or a negative value on error */ -int ltfs_sync_index(char *reason, bool index_locking, struct ltfs_volume *vol) +int ltfs_sync_index(char *reason, bool index_locking, enum ltfs_index_type type, struct ltfs_volume *vol) { int ret = 0, ret_r = 0; bool dirty; @@ -3568,6 +3692,10 @@ int ltfs_sync_index(char *reason, bool index_locking, struct ltfs_volume *vol) bool dp_index_file_end, ip_index_file_end; char *bc_print = NULL; +#ifndef FORMAT_SPEC25 + type = LTFS_FULL_INDEX; +#endif + start: ret = ltfs_get_partition_readonly(ltfs_dp_id(vol), vol); if (ret < 0 && ret != -LTFS_LESS_SPACE) @@ -3622,7 +3750,7 @@ int ltfs_sync_index(char *reason, bool index_locking, struct ltfs_volume *vol) releasewrite_mrsw(&vol->lock); return ret; } - ret = ltfs_write_index(partition, reason, vol); + ret = ltfs_write_index(partition, reason, type, vol); if (IS_WRITE_PERM(-ret) && partition == ltfs_dp_id(vol)) { /* * TODO: Need to determine the last record on DP of the tape and cleanup @@ -3675,13 +3803,14 @@ int ltfs_sync_index(char *reason, bool index_locking, struct ltfs_volume *vol) * @param vol LTFS volume * @param partition partition to traverse * @param gen generation to search. 0 to search all + * @param skip_dir skip parsing directory * @param func call back function to call when an index is find. NULL not to call * @param list this pointer is specified as 3rd arguments of call back function * @param priv private data for call back function * @return 0 on success or a negative value on error */ int ltfs_traverse_index_no_eod(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void* priv) + bool skip_dir, f_index_found func, void **list, void* priv) { int ret, func_ret; @@ -3694,7 +3823,7 @@ int ltfs_traverse_index_no_eod(struct ltfs_volume *vol, char partition, unsigned while (true) { ltfs_index_free(&vol->index); ltfs_index_alloc(&vol->index, vol); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, skip_dir, vol); if (ret < 0 && ret != -LTFS_UNSUPPORTED_INDEX_VERSION) { ltfsmsg(LTFS_ERR, 17075E, 'N', (int)vol->device->position.block, partition); return ret; @@ -3748,13 +3877,14 @@ int ltfs_traverse_index_no_eod(struct ltfs_volume *vol, char partition, unsigned * @param vol LTFS volume * @param partition partition to traverse * @param gen generation to search. 0 to search all + * @param skip_dir skip parsing directory * @param func call back function to call when an index is find. NULL not to call * @param list this pointer is specified as 3rd arguments of call back function * @param priv private data for call back function * @return 0 on success or a negative value on error */ int ltfs_traverse_index_forward(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void* priv) + bool skip_dir, f_index_found func, void **list, void* priv) { int ret, func_ret; struct tape_offset last_index; @@ -3777,7 +3907,7 @@ int ltfs_traverse_index_forward(struct ltfs_volume *vol, char partition, unsigne while (last_index.block >= vol->device->position.block) { ltfs_index_free(&vol->index); ltfs_index_alloc(&vol->index, vol); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, skip_dir, vol); if (ret < 0 && ret != -LTFS_UNSUPPORTED_INDEX_VERSION) { ltfsmsg(LTFS_ERR, 17075E, 'F', (int)vol->device->position.block, partition); return ret; @@ -3832,12 +3962,13 @@ int ltfs_traverse_index_forward(struct ltfs_volume *vol, char partition, unsigne * @param vol LTFS volume * @param partition partition to traverse * @param gen generation to search. 0 to search all + * @param skip_dir skip parsing directory * @param func call back function to call when an index is find. NULL not to call * @param list this pointer is specified as 3rd arguments of call back function * @return 0 on success or a negative value on error */ int ltfs_traverse_index_backward(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void* priv) + bool skip_dir, f_index_found func, void **list, void* priv) { int ret, func_ret; @@ -3853,7 +3984,7 @@ int ltfs_traverse_index_backward(struct ltfs_volume *vol, char partition, unsign ltfs_index_free(&vol->index); ltfs_index_alloc(&vol->index, vol); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, skip_dir, vol); if (ret < 0 && ret != -LTFS_UNSUPPORTED_INDEX_VERSION) { ltfsmsg(LTFS_ERR, 17075E, 'B', (int)vol->device->position.block, partition); return ret; @@ -4024,7 +4155,7 @@ static int _ltfs_detect_final_rec_dp(struct ltfs_volume *vol, struct tc_position INTERRUPTED_RETURN(); ltfsmsg(LTFS_INFO, 17120I, "DP", (unsigned long long)seekpos.partition, (unsigned long long)seekpos.block); - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, false, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 17121E, "DP", ret); return ret; @@ -4200,7 +4331,7 @@ int ltfs_recover_eod(struct ltfs_volume *vol) * In index partition, Index will be overwritten. * Read an index only current partition is DP. */ - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, false, vol); if (ret < 0) return ret; } @@ -4379,7 +4510,7 @@ int ltfs_print_device_list(struct tape_ops *ops) } } - ret = 0; + //ret = 0; return ret; } diff --git a/src/libltfs/ltfs.h b/src/libltfs/ltfs.h index 5a792f55..c60f5552 100644 --- a/src/libltfs/ltfs.h +++ b/src/libltfs/ltfs.h @@ -48,6 +48,10 @@ ** IBM Almaden Research Center ** lucasvr@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ #ifndef __ltfs_h__ @@ -149,16 +153,34 @@ struct device_data; #define LTFS_LABEL_VERSION_MIN MAKE_LTFS_VERSION(1,0,0) /* Min supported label version */ #define LTFS_LABEL_VERSION_MAX MAKE_LTFS_VERSION(2,99,99) /* Max supported label version */ + +#ifdef FORMAT_SPEC25 +#define LTFS_LABEL_VERSION MAKE_LTFS_VERSION(2,5,0) /* Written label version */ +#define LTFS_LABEL_VERSION_STR "2.5.0" /* Label version string */ +#else #define LTFS_LABEL_VERSION MAKE_LTFS_VERSION(2,4,0) /* Written label version */ #define LTFS_LABEL_VERSION_STR "2.4.0" /* Label version string */ +#endif #define LTFS_INDEX_VERSION_MIN MAKE_LTFS_VERSION(1,0,0) /* Min supported index version */ #define LTFS_INDEX_VERSION_MAX MAKE_LTFS_VERSION(2,99,99) /* Max supported index version */ + +#ifdef FORMAT_SPEC25 +#define LTFS_INDEX_VERSION MAKE_LTFS_VERSION(2,5,0) /* Written index version */ +#define LTFS_INDEX_VERSION_STR "2.5.0" /* Index version string */ +#else #define LTFS_INDEX_VERSION MAKE_LTFS_VERSION(2,4,0) /* Written index version */ #define LTFS_INDEX_VERSION_STR "2.4.0" /* Index version string */ +#endif #define INDEX_MAX_COMMENT_LEN 65536 /* Maximum comment field length (per LTFS Format) */ +enum ltfs_index_type { + LTFS_INDEX_AUTO = 0, /**< Select index type to write based on specified options */ + LTFS_FULL_INDEX, /**< Forcibly write full index */ + LTFS_INCREMENTAL_INDEX /**< Forcibly write incremental index */ +}; + #define LTFS_NO_BARCODE "NO_BARCODE" #ifndef __APPLE_MAKEFILE__ @@ -334,7 +356,7 @@ struct dentry { /* Take the iosched_lock before accessing iosched_priv. */ void *iosched_priv; /**< I/O scheduler private data. */ - struct name_list *child_list; /* for hash search */ + struct name_list *child_list; /**< Child list for hash search */ }; struct tape_attr { @@ -358,7 +380,7 @@ typedef enum mam_advisory_lock_status { PWE_MAM_IP = 5, /* Single write perm on IP (Set VOL_IP_PERM__ERR) */ PWE_MAM_BOTH = 6, /* Double write perm (Set both VOL_DP_PERM_ERR and VOL_IP_PERM_ERR) */ NOLOCK_MAM = 128, /* From HPE */ -} mam_lockval; +} mam_lockval_t; #define IS_SINGLE_WRITE_PERM(stat) (stat == PWE_MAM || (stat == PWE_MAM_DP || stat == PWE_MAM_IP) ) #define IS_DOUBLE_WRITE_PERM(stat) (stat == PWE_MAM_BOTH) @@ -381,9 +403,10 @@ enum volumelock_status { }; enum volume_mount_type { - MOUNT_NORMAL = 0, /**< Normal mount */ - MOUNT_ROLLBACK, /**< Roll back mount */ - MOUNT_ERR_TAPE, /**< Mount write perm tape */ + MOUNT_NORMAL = 0, /**< Normal mount */ + MOUNT_ROLLBACK, /**< Roll back mount */ + MOUNT_ROLLBACK_META, /**< Roll back mount only with meta-data */ + MOUNT_ERR_TAPE, /**< Mount write perm tape */ }; #define VOL_WRITE_PERM_MASK (0xE0) @@ -399,7 +422,8 @@ struct ltfs_volume { struct tc_coherency dp_coh; /**< Data partition coherency info */ struct ltfs_label *label; /**< Information from the partition labels */ struct ltfs_index *index; /**< Current cartridge index */ - char *index_cache_path; /**< File name of on-disk index cache */ + char *index_cache_path_w; /**< File name of on-disk index cache update at writing an index */ + char *index_cache_path_r; /**< File name of on-disk index cache update at parsing an index */ /* Opaque handles to higher-level structures */ void *iosched_handle; /**< Handle to the I/O scheduler state */ @@ -449,9 +473,9 @@ struct ltfs_volume { char *mountpoint; /**< Store mount point for Live Link (SDE) */ size_t mountpoint_len; /**< Store mount point path length (SDE) */ struct tape_attr *t_attr; /**< Tape Attribute data */ - mam_lockval lock_status; /**< Total volume lock status from t_attr->vollock and index->vollock */ + mam_lockval_t lock_status; /**< Total volume lock status from t_attr->vollock and index->vollock */ struct ltfs_timespec first_locate; /**< Time to first locate */ - int file_open_count; /**< Number of opened files */ + int file_open_count; /**< Number of opened files */ const char *work_directory; @@ -508,10 +532,10 @@ struct ltfs_index { struct index_criteria index_criteria; /**< Active index criteria */ struct dentry *root; /**< The directory tree */ - ltfs_mutex_t rename_lock; /**< Controls name tree access during renames */ + ltfs_mutex_t rename_lock; /**< Controls name tree access during renames */ /* Update tracking */ - ltfs_mutex_t dirty_lock; /**< Controls access to the update tracking bits */ + ltfs_mutex_t dirty_lock; /**< Controls access to the update tracking bits */ bool dirty; /**< Set on metadata update, cleared on write to tape */ bool atime_dirty; /**< Set on atime update, cleared on write to tape */ bool use_atime; /**< Set if atime updates should make the index dirty */ @@ -530,7 +554,13 @@ struct ltfs_index { size_t symerr_count; /**< Number of conflicted symlink dentries */ struct dentry **symlink_conflict; /**< symlink/extent conflicted dentries */ - mam_lockval vollock; /**< volume lock status on index */ + mam_lockval_t vollock; /**< volume lock status on index */ + + /* Incremental index */ + uint64_t full_index_interval; /**< Number of indexes between full indexes */ + uint64_t full_index_to_go; /**< How many incremental index shall be written to the next full index */ + struct tape_offset selfptr_inc; /**< self-pointer of previous incremental index (to prior generation on data partition) */ + struct tape_offset backptr_inc; /**< back pointer of previous incremental index (to prior generation on data partition) */ }; struct ltfs_direntry { @@ -652,6 +682,7 @@ bool ltfs_get_criteria_allow_update(struct ltfs_volume *vol); int ltfs_start_mount(bool trial, struct ltfs_volume *vol); int ltfs_mount(bool force_full, bool deep_recovery, bool recover_extra, bool recover_symlink, unsigned short gen, struct ltfs_volume *vol); +int ltfs_mount_indexfile(char* filename, bool label_check, struct ltfs_volume *vol); int ltfs_unmount(char *reason, struct ltfs_volume *vol); void ltfs_dump_tree_unlocked(struct ltfs_index *index); void ltfs_dump_tree(struct ltfs_volume *vol); @@ -695,14 +726,14 @@ char ltfs_dp_id(struct ltfs_volume *vol); char ltfs_ip_id(struct ltfs_volume *vol); const char *ltfs_get_volume_uuid(struct ltfs_volume *vol); -int ltfs_sync_index(char *reason, bool index_locking, struct ltfs_volume *vol); +int ltfs_sync_index(char *reason, bool index_locking, enum ltfs_index_type type, struct ltfs_volume *vol); int ltfs_traverse_index_forward(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void *priv); + bool skip_dir, f_index_found func, void **list, void *priv); int ltfs_traverse_index_backward(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void *priv); + bool skip_dir, f_index_found func, void **list, void *priv); int ltfs_traverse_index_no_eod(struct ltfs_volume *vol, char partition, unsigned int gen, - f_index_found func, void **list, void *priv); + bool skip_dir, f_index_found func, void **list, void *priv); int ltfs_check_eod_status(struct ltfs_volume *vol); int ltfs_recover_eod(struct ltfs_volume *vol); int ltfs_release_medium(struct ltfs_volume *vol); diff --git a/src/libltfs/ltfs_error.h b/src/libltfs/ltfs_error.h index c762520e..270556b7 100644 --- a/src/libltfs/ltfs_error.h +++ b/src/libltfs/ltfs_error.h @@ -518,4 +518,16 @@ #define MKLTFS_USAGE_SYNTAX_ERROR PROG_USAGE_SYNTAX_ERROR /* Wrong argument */ #define MKLTFS_CANCELED_BY_USER PROG_CANCELED_BY_USER /* Canceled by user */ +/* Status code for ltfsindextool + * The maximum return code from the program is 0xFF. + */ +#define INDEXTOOL_NO_ERRORS PROG_NO_ERRORS /* No error and the cartridge is not modified */ +#define INDEXTOOL_CORRECTED PROG_TREAT_SUCCESS /* Recover correctly, the cartridge is modified */ +#define INDEXTOOL_REBOOT_REQUIRED PROG_REBOOT_REQUIRED /* Reboot required */ +#define INDEXTOOL_UNCORRECTED PROG_UNCORRECTED /* Cannot recover, the cartridge is modified */ +#define INDEXTOOL_OPERATIONAL_ERROR PROG_OPERATIONAL_ERROR /* Get device error while processing, the cartridge may be modified */ +#define INDEXTOOL_USAGE_SYNTAX_ERROR PROG_USAGE_SYNTAX_ERROR /* Wrong argument */ +#define INDEXTOOL_CANCELED_BY_USER PROG_CANCELED_BY_USER /* Canceled by user */ +#define INDEXTOOL_SHARED_LIB_ERROR PROG_SHARED_LIB_ERROR /* Library error */ + #endif /* __ltfs_error_h__ */ diff --git a/src/libltfs/ltfs_fsops.c b/src/libltfs/ltfs_fsops.c index 7b2aa4d4..3469eeeb 100644 --- a/src/libltfs/ltfs_fsops.c +++ b/src/libltfs/ltfs_fsops.c @@ -66,6 +66,7 @@ #include "dcache.h" #include "pathname.h" #include "index_criteria.h" +#include "inc_journal.h" #include "arch/time_internal.h" int ltfs_fsops_open(const char *path, bool open_write, bool use_iosched, struct dentry **d, @@ -367,9 +368,10 @@ int ltfs_fsops_create(const char *path, bool isdir, bool readonly, bool overwrit if (! isdir) ++vol->index->file_count; ltfs_set_index_dirty(false, false, vol->index); + incj_create(path_norm, d, vol); d->dirty = true; ltfs_mutex_unlock(&vol->index->dirty_lock); - vol->file_open_count ++; + vol->file_open_count++; *dentry = d; ret = 0; @@ -522,8 +524,12 @@ int ltfs_fsops_unlink(const char *path, ltfs_file_id *id, struct ltfs_volume *vo releasewrite_mrsw(&d->meta_lock); ltfs_mutex_lock(&vol->index->dirty_lock); - if (! d->isdir) + if (d->isdir) { + incj_rmdir(path_norm, d, vol); + } else { + incj_rmfile(path_norm, d, vol); --vol->index->file_count; + } ltfs_set_index_dirty(false, false, vol->index); ltfs_mutex_unlock(&vol->index->dirty_lock); @@ -868,6 +874,16 @@ int ltfs_fsops_rename(const char *from, const char *to, ltfs_file_id *id, struct fromdentry->dirty = true; + /* Process the incremental journal */ + if (fromdentry->isdir) + incj_rmdir(from_norm_copy, fromdentry, vol); + else + incj_rmfile(from_norm_copy, fromdentry, vol); + + fs_split_path(to_norm_copy, &to_filename_incj, strlen(to_norm_copy) + 1); + incj_create(to_norm_copy, fromdentry, vol); + + /* Release dentry of source */ if (! iosched_initialized(vol)) fs_release_dentry_unlocked(fromdentry); else @@ -1565,7 +1581,7 @@ int ltfs_fsops_utimens(struct dentry *d, const struct ltfs_timespec ts[2], struc d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[0].tv_sec); get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, true, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (d->modify_time.tv_sec != ts[1].tv_sec || d->modify_time.tv_nsec != ts[1].tv_nsec) { d->modify_time = ts[1]; @@ -1575,7 +1591,7 @@ int ltfs_fsops_utimens(struct dentry *d, const struct ltfs_timespec ts[2], struc d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[1].tv_sec); get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, false, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (dcache_initialized(vol)) dcache_flush(d, FLUSH_METADATA, vol); @@ -1642,7 +1658,7 @@ int ltfs_fsops_utimens_all(struct dentry *d, const struct ltfs_timespec ts[4], s d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[3].tv_sec); isctime=true; ltfs_set_index_dirty(true, false, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (ts[0].tv_sec != 0 || ts[0].tv_nsec != 0) { d->access_time = ts[0]; @@ -1652,7 +1668,7 @@ int ltfs_fsops_utimens_all(struct dentry *d, const struct ltfs_timespec ts[4], s d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[0].tv_sec); if(!isctime) get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, true, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (ts[1].tv_sec != 0 || ts[1].tv_nsec != 0) { d->modify_time = ts[1]; @@ -1662,7 +1678,7 @@ int ltfs_fsops_utimens_all(struct dentry *d, const struct ltfs_timespec ts[4], s d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[1].tv_sec); if(!isctime) get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, false, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (ts[2].tv_sec != 0 || ts[2].tv_nsec != 0) { d->creation_time = ts[2]; @@ -1672,7 +1688,7 @@ int ltfs_fsops_utimens_all(struct dentry *d, const struct ltfs_timespec ts[4], s d->platform_safe_name, (unsigned long long)d->uid, (unsigned long long)ts[2].tv_sec); if(!isctime) get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, false, vol->index); - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); } if (dcache_initialized(vol)) @@ -1684,7 +1700,6 @@ int ltfs_fsops_utimens_all(struct dentry *d, const struct ltfs_timespec ts[4], s return 0; } - int ltfs_fsops_set_readonly(struct dentry *d, bool readonly, struct ltfs_volume *vol) { int ret; @@ -1710,6 +1725,7 @@ int ltfs_fsops_set_readonly(struct dentry *d, bool readonly, struct ltfs_volume d->readonly = readonly; get_current_timespec(&d->change_time); ltfs_set_index_dirty(true, false, vol->index); + ltfs_set_dentry_dirty(d, vol); if (dcache_initialized(vol)) dcache_flush(d, FLUSH_METADATA, vol); } @@ -1798,6 +1814,10 @@ ssize_t ltfs_fsops_read(struct dentry *d, char *buf, size_t count, off_t offset, if (d->isdir) return -LTFS_ISDIRECTORY; + if (vol->mount_type == MOUNT_ROLLBACK_META) { + return -LTFS_DEVICE_UNREADY; + } + if (iosched_initialized(vol)) ret = iosched_read(d, buf, count, offset, vol); else @@ -2092,7 +2112,11 @@ int ltfs_fsops_volume_sync(char *reason, struct ltfs_volume *vol) if (ret < 0) return ret; - ret = ltfs_sync_index(reason, true, vol); + ltfs_mutex_lock(&vol->index->dirty_lock); + ltfs_set_commit_message_reason_unlocked(reason, vol); + ltfs_mutex_unlock(&vol->index->dirty_lock); + + ret = ltfs_sync_index(reason, true, LTFS_INDEX_AUTO, vol); return ret; } diff --git a/src/libltfs/ltfs_fsops_raw.c b/src/libltfs/ltfs_fsops_raw.c index de61fa12..87086bb3 100644 --- a/src/libltfs/ltfs_fsops_raw.c +++ b/src/libltfs/ltfs_fsops_raw.c @@ -398,7 +398,7 @@ int _ltfs_fsraw_add_extent_unlocked(struct dentry *d, struct extent_info *ext, b * No need to mark at this time but reserve this value for fueture release */ d->extents_dirty = true; - d->dirty = true; + ltfs_set_dentry_dirty(d, vol); releasewrite_mrsw(&d->meta_lock); ltfs_set_index_dirty(true, false, vol->index); @@ -793,12 +793,12 @@ int ltfs_fsraw_truncate(struct dentry *d, off_t length, struct ltfs_volume *vol) d->realsize = new_realsize; get_current_timespec(&d->modify_time); d->change_time = d->modify_time; + ltfs_set_dentry_dirty(d, vol); releasewrite_mrsw(&d->meta_lock); releasewrite_mrsw(&d->contents_lock); ltfs_set_index_dirty(true, false, vol->index); - d->dirty = true; releaseread_mrsw(&vol->lock); return 0; diff --git a/src/libltfs/ltfs_internal.c b/src/libltfs/ltfs_internal.c index 6059a0fc..2af5b28c 100644 --- a/src/libltfs/ltfs_internal.c +++ b/src/libltfs/ltfs_internal.c @@ -44,6 +44,10 @@ ** IBM Almaden Research Center ** bbiskebo@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ @@ -61,6 +65,7 @@ #include "iosched.h" #include "ltfs_fsops.h" #include "xattr.h" +#include "inc_journal.h" /** * Allocate an empty LTFS index. @@ -384,11 +389,13 @@ int ltfs_read_one_label(tape_partition_t partition, struct ltfs_label *label, * This function does not read over another file mark * @param eod_pos EOD position for current partition, or 0 to assume that EOD will not be * encountered during parsing. + * @param recover_symlink recover symlink conflict + * @param skip_dir skip parsing directory * @param vol the volume * @return 0 on success, 1 if index file does not end with a file mark (but is otherwise valid), * or a negative value on error. */ -int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, struct ltfs_volume *vol) +int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, bool skip_dir, struct ltfs_volume *vol) { int ret, ret_sym; struct tc_position pos; @@ -410,7 +417,7 @@ int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, struct ltfs_volume * } /* Parse and validate the schema */ - ret = xml_schema_from_tape(eod_pos, vol); + ret = xml_schema_from_tape(eod_pos, skip_dir, vol); if ( vol->index->symerr_count ) { if ( recover_symlink ) { ret_sym = ltfs_split_symlink( vol ); @@ -473,6 +480,49 @@ int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, struct ltfs_volume * return end_fm ? 0 : 1; } +/** + * Read an index file from file, storing the result in the given volume. + * + * @param filename file name of index + * @param recover_symlink recover symlink conflict + * @param vol the volume + * @return 0 on success, or a negative value on error. + */ +int ltfs_read_indexfile(char* filename, bool recover_symlink, struct ltfs_volume *vol) +{ + int ret, ret_sym; + + ltfs_index_free(&vol->index); + ret = ltfs_index_alloc(&vol->index, vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 11297E, ret); + return ret; + } + + /* Parse and validate the schema */ + ret = xml_schema_from_file(filename, vol->index, vol); + if ( vol->index->symerr_count ) { + if ( recover_symlink ) { + ret_sym = ltfs_split_symlink( vol ); + if (ret_sym < 0) { + ret = ret_sym; + } else if (ret == -LTFS_SYMLINK_CONFLICT) { + ret = 0; + } + } else { + ltfsmsg(LTFS_ERR, 11321E); + } + free( vol->index->symlink_conflict ); + vol->index->symerr_count = 0; + } + + if (ret < 0) { + ltfsmsg(LTFS_WARN, 11194W, ret); + } + + return ret; +} + /** * Returns true iff a given char corresponds to a valid logical partition ID. */ @@ -540,7 +590,7 @@ int ltfs_seek_index(char partition, tape_block_t *eod_pos, tape_block_t *index_e /* try to read an index file */ check_err(tape_spacefm(vol->device, 1), 11202E, out); - ret = ltfs_read_index(*eod_pos, recover_symlink, vol); + ret = ltfs_read_index(*eod_pos, recover_symlink, false, vol); if (ret < 0) { /* no index file found: go back 2 file marks and try again */ ltfsmsg(LTFS_DEBUG, 11204D); @@ -704,7 +754,7 @@ int _ltfs_check_pointers(struct ltfs_index *ip_index, struct ltfs_index *dp_inde ret = tape_seek(vol->device, &seekpos); if (ret < 0) return ret; - ret = ltfs_read_index(0, false, vol); + ret = ltfs_read_index(0, false, true, vol); if (ret < 0) return ret; dp_backptr = vol->index->backptr.block; @@ -982,6 +1032,68 @@ int _ltfs_make_lost_found(tape_block_t ip_eod, tape_block_t dp_eod, return 0; } +/** + * Find the appropriate append address for a partition at which calling ltfs_write_index() will + * correctly restore consistency to the partition. + * + * @param vol The ltfs volume to work on + * @param partid The partition to examine + * @param block The starting block of the final index on the partition + * @param fix allow simple fixes to make the volume consistent? + * @return 0 if eod is the appropriate append address, >0 for the absolute block address to append + * at, <0 on error + */ +static int _ltfs_find_append_blk_after_idx(struct ltfs_volume *vol, char partid, tape_block_t block, bool fix) { + unsigned int n_fm_after = 0; + struct tc_position idx_pos; + struct tc_position final_fm_pos; + int ret = 0; + + idx_pos.partition = ltfs_part_id2num(partid, vol); + idx_pos.block = block; + ret = tape_seek(vol->device, &idx_pos); + if (ret != 0) { + if (partid == vol->label->partid_ip) + ltfsmsg(LTFS_ERR, 11023E); + else { + ltfsmsg(LTFS_ERR, 11020E); + } + goto out; + } + while(ret == 0) { + ret = tape_spacefm(vol->device, 1); + if (ret == 0) { + tape_update_position(vol->device, &final_fm_pos); + n_fm_after++; + } else { + if (ret != -EDEV_EOD_DETECTED) { + goto out; + } + } + } + ret = 0; + switch (n_fm_after){ + case 0: + case 1: + /* Append block does not need to be altered */ + break; + case 2: + /* (index | data | ... eod) - unexpected fm after data, possible incomplete index */ + if (fix) { + ret = final_fm_pos.block-1; + } else { + ret = -LTFS_OP_TO_INV; + } + break; + default: + /* (index | data | ??? | ... eod) - invalid format*/ + ret = -LTFS_OP_TO_INV; + break; + } +out: + return ret; +} + /** * Check a volume for physical consistency. This should be called when there is some doubt about * the validity of the MAM parameters; it reads index files from both partitions and verifies @@ -993,7 +1105,8 @@ int _ltfs_make_lost_found(tape_block_t ip_eod, tape_block_t dp_eod, * * @param fix allow simple fixes to make the tape consistent? * Here, simple means writing an additional logically unmodified copy - * or copies of an index file already present on the tape. + * or copies of an index file already present on the tape. Also + * allows truncating incomplete index at the end of the partition. * @param deep Allow fancy recovery procedures? In particular, this flag enables recovery in the * case where extra blocks (after the last index file on a partition) are found on the * tape. The nature of this recovery is controlled by the recover_extra flag. @@ -1121,8 +1234,28 @@ int ltfs_check_medium(bool fix, bool deep, bool recover_extra, bool recover_syml if (ret < 0) goto out_unlock; - /* Set append position for index partition. */ - if (ip_have_index && ! ip_blocks_after) { + /* Set append position for data partition to end of trailing data. */ + if (dp_have_index && dp_blocks_after) { + ret = _ltfs_find_append_blk_after_idx(vol, vol->label->partid_dp, dp_index->selfptr.block, fix); + if (ret < 0) { + goto out_unlock; + } else { + dp_eod = ret; + check_err(tape_set_append_position(vol->device, dp_num, dp_eod), + 11222E, out_unlock); + } + } + + /* Set append position for index partition to end of trailing data or preceding data */ + if (ip_have_index && ip_blocks_after) { + ret = _ltfs_find_append_blk_after_idx(vol, vol->label->partid_ip, ip_index->selfptr.block, fix); + if (ret <0) { + goto out_unlock; + } else { + ip_eod = ret; + check_err(tape_set_append_position(vol->device, ip_num, ip_eod), + 11222E, out_unlock); + } check_err(tape_set_append_position(vol->device, ip_num, ip_index->selfptr.block - 1), 11222E, out_unlock); } @@ -1190,13 +1323,14 @@ int ltfs_check_medium(bool fix, bool deep, bool recover_extra, bool recover_syml } } /* write to data partition if it doesn't end in an index file */ + ltfs_set_commit_message_reason(SYNC_RECOVERY, vol); if (! dp_have_index || dp_blocks_after) { ltfsmsg(LTFS_INFO, 17259I, "DP", vol->index->selfptr.partition, (unsigned long long)vol->index->selfptr.block); - ret = ltfs_write_index(vol->label->partid_dp, SYNC_RECOVERY, vol); + ret = ltfs_write_index(vol->label->partid_dp, SYNC_RECOVERY, LTFS_FULL_INDEX, vol); } if (!ret) { ltfsmsg(LTFS_INFO, 17259I, "IP", vol->index->selfptr.partition, (unsigned long long)vol->index->selfptr.block); - ltfs_write_index(vol->label->partid_ip, SYNC_RECOVERY, vol); + ltfs_write_index(vol->label->partid_ip, SYNC_RECOVERY, LTFS_FULL_INDEX, vol); } } else { ltfsmsg(LTFS_ERR, 11231E); @@ -1274,12 +1408,15 @@ int ltfs_write_index_conditional(char partition, struct ltfs_volume *vol) CHECK_ARG_NULL(vol, -LTFS_NULL_ARG); - if (partition == ltfs_ip_id(vol) && ! vol->ip_index_file_end) - ret = ltfs_write_index(partition, SYNC_CASCHE_PRESSURE, vol); - else if (partition == ltfs_dp_id(vol) && + if (partition == ltfs_ip_id(vol) && ! vol->ip_index_file_end) { + ltfs_set_commit_message_reason(SYNC_CASCHE_PRESSURE, vol); + ret = ltfs_write_index(partition, SYNC_CASCHE_PRESSURE, LTFS_FULL_INDEX, vol); + } else if (partition == ltfs_dp_id(vol) && (! vol->dp_index_file_end || - (vol->ip_index_file_end && vol->index->selfptr.partition == ltfs_ip_id(vol)))) - ret = ltfs_write_index(partition, SYNC_CASCHE_PRESSURE, vol); + (vol->ip_index_file_end && vol->index->selfptr.partition == ltfs_ip_id(vol)))) { + ltfs_set_commit_message_reason(SYNC_CASCHE_PRESSURE, vol); + ret = ltfs_write_index(partition, SYNC_CASCHE_PRESSURE, LTFS_FULL_INDEX, vol); + } return ret; } @@ -1395,3 +1532,22 @@ int ltfs_split_symlink(struct ltfs_volume *vol) free(path); return ret; } + +int ltfs_set_dentry_dirty(struct dentry *d, struct ltfs_volume *vol) +{ + int ret = 0; + char *full_path = NULL; + + if (!d->dirty) { + ret = ltfs_build_fullpath(&full_path, d); + if (!ret) { + incj_modify(full_path, d, vol); + } else { + vol->journal_err = true; + } + } + + d->dirty = true; + + return ret; +} diff --git a/src/libltfs/ltfs_internal.h b/src/libltfs/ltfs_internal.h index da0cfa47..71f7df16 100644 --- a/src/libltfs/ltfs_internal.h +++ b/src/libltfs/ltfs_internal.h @@ -44,6 +44,10 @@ ** IBM Almaden Research Center ** bbiskebo@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ @@ -70,7 +74,8 @@ int ltfs_check_medium(bool fix, bool deep, bool recover_extra, bool recover_syml int ltfs_read_labels(bool trial, struct ltfs_volume *vol); int ltfs_read_one_label(tape_partition_t partition, struct ltfs_label *label, struct ltfs_volume *vol); -int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, struct ltfs_volume *vol); +int ltfs_read_index(uint64_t eod_pos, bool recover_symlink, bool skip_dir, struct ltfs_volume *vol); +int ltfs_read_indexfile(char* filename, bool recover_symlink, struct ltfs_volume *vol); int ltfs_update_cart_coherency(struct ltfs_volume *vol); int ltfs_write_index_conditional(char partition, struct ltfs_volume *vol); @@ -80,6 +85,7 @@ int ltfs_seek_index(char partition, tape_block_t *eod_pos, tape_block_t *index_e bool *fm_after, bool *blocks_after, bool recover_symlink, struct ltfs_volume *vol); void _ltfs_last_ref(struct dentry *d, tape_block_t *dp_last, tape_block_t *ip_last, struct ltfs_volume *vol); -int ltfs_split_symlink( struct ltfs_volume *vol ); +int ltfs_split_symlink(struct ltfs_volume *vol); +int ltfs_set_dentry_dirty(struct dentry *d, struct ltfs_volume *vol); #endif /* __ltfs_internal_h__ */ diff --git a/src/libltfs/pathname.c b/src/libltfs/pathname.c index 19abc321..9941d0ad 100644 --- a/src/libltfs/pathname.c +++ b/src/libltfs/pathname.c @@ -57,7 +57,7 @@ #include #include #include -#ifdef ICU6x +#ifdef USE_UNORM2 #include #else #include @@ -67,7 +67,7 @@ #include #include #include -#ifdef ICU6x +#ifdef USE_UNORM2 #include #else #include @@ -649,13 +649,13 @@ int _pathname_foldcase_icu(const UChar *src, UChar **dest) } /** - * ICU5x/ICU6x handle: gets a reference to a singleton unorm2 object (ICU6x) or to a constant + * ICU5x/USE_UNORM2 handle: gets a reference to a singleton unorm2 object (USE_UNORM2) or to a constant * value that we can use to differentiate between NFC and NFD modes (ICU5x). Neither should be * freed after use. */ static inline void *_unorm_handle(bool nfc, UErrorCode *err) { -#ifdef ICU6x +#ifdef USE_UNORM2 *err = U_ZERO_ERROR; return (void *) unorm2_getInstance(NULL, "nfc", nfc ? UNORM2_COMPOSE : UNORM2_DECOMPOSE, err); #else @@ -666,7 +666,7 @@ static inline void *_unorm_handle(bool nfc, UErrorCode *err) static inline UNormalizationCheckResult _unorm_quickCheck(void *handle, const UChar *src, UChar **dest, UErrorCode *err) { *err = U_ZERO_ERROR; -#ifdef ICU6x +#ifdef USE_UNORM2 const UNormalizer2 *n2 = (const UNormalizer2 *) handle; return unorm2_quickCheck(n2, src, -1, err); #else @@ -678,7 +678,7 @@ static inline UNormalizationCheckResult _unorm_quickCheck(void *handle, const UC static inline int32_t _unorm_normalize(void *handle, const UChar *src, UChar **dest, int32_t len, UErrorCode *err) { *err = U_ZERO_ERROR; -#ifdef ICU6x +#ifdef USE_UNORM2 const UNormalizer2 *n2 = (const UNormalizer2 *) handle; return unorm2_normalize(n2, src, -1, dest ? *dest : NULL, len, err); #else diff --git a/src/libltfs/periodic_sync.c b/src/libltfs/periodic_sync.c index f0ac14e4..9aa6ef39 100644 --- a/src/libltfs/periodic_sync.c +++ b/src/libltfs/periodic_sync.c @@ -43,6 +43,7 @@ ** AUTHOR: Atsushi Abe ** IBM Yamato, Japan ** PISTE@jp.ibm.com +** ************************************************************************************* */ @@ -99,6 +100,12 @@ ltfs_thread_return periodic_sync_thread(void* data) if (! priv->keepalive) break; + if (priv->vol->mount_type == MOUNT_ROLLBACK || + priv->vol->mount_type == MOUNT_ROLLBACK_META) { + /* Never call sync on R/O mount */ + continue; + } + ltfs_request_trace(FUSE_REQ_ENTER(REQ_SYNC), 0, 0); ltfsmsg(LTFS_DEBUG, 17067D, "Sync-by-Time"); @@ -108,7 +115,9 @@ ltfs_thread_return periodic_sync_thread(void* data) ltfsmsg(LTFS_WARN, 17063W, __FUNCTION__); } - ret = ltfs_sync_index(SYNC_PERIODIC, true, priv->vol); + ltfs_set_commit_message_reason(SYNC_PERIODIC, priv->vol); + + ret = ltfs_sync_index(SYNC_PERIODIC, true, LTFS_INDEX_AUTO, priv->vol); if (ret < 0) { ltfsmsg(LTFS_INFO, 11030I, ret); priv->keepalive = false; diff --git a/src/libltfs/xattr.c b/src/libltfs/xattr.c index 0cd89575..c8bb80cf 100644 --- a/src/libltfs/xattr.c +++ b/src/libltfs/xattr.c @@ -66,6 +66,7 @@ #include "pathname.h" #include "tape.h" #include "ltfs_internal.h" +#include "inc_journal.h" #include "arch/time_internal.h" /* Helper functions for formatting virtual EA output */ @@ -1443,15 +1444,6 @@ int xattr_set(struct dentry *d, const char *name, const char *value, size_t size } } - if (!strcmp(name, "ltfs.mediaPool.name")) { - ret = tape_set_media_pool_info(vol, value, size, true); - if (ret < 0) { - releasewrite_mrsw(&d->meta_lock); - goto out_unlock; - } - write_idx = true; - } - /* Set extended attribute */ ret = xattr_do_set(d, name, value, size, xattr); if (ret < 0) { @@ -1470,13 +1462,16 @@ int xattr_set(struct dentry *d, const char *name, const char *value, size_t size } get_current_timespec(&d->change_time); - releasewrite_mrsw(&d->meta_lock); - d->dirty = true; + ltfs_set_index_dirty(true, false, vol->index); + ltfs_set_dentry_dirty(d, vol); - if (write_idx) - ret = ltfs_sync_index(SYNC_EA, false, vol); - else + releasewrite_mrsw(&d->meta_lock); + + if (write_idx) { + ltfs_set_commit_message_reason(SYNC_EA, vol); + ret = ltfs_sync_index(SYNC_EA, false, LTFS_INDEX_AUTO, vol); + } else ret = 0; out_unlock: @@ -1516,6 +1511,12 @@ int xattr_get(struct dentry *d, const char *name, char *value, size_t size, /* Try to get a virtual xattr first. */ if (_xattr_is_virtual(d, name, vol)) { + + if (vol->mount_type == MOUNT_ROLLBACK_META) { + _xattr_unlock_dentry(name, false, d, vol); + return -LTFS_DEVICE_UNREADY; + } + ret = _xattr_get_virtual(d, value, size, name, vol); if (ret == -LTFS_DEVICE_FENCED) { _xattr_unlock_dentry(name, false, d, vol); @@ -1726,8 +1727,8 @@ int xattr_remove(struct dentry *d, const char *name, struct ltfs_volume *vol) ltfsmsg(LTFS_INFO, 17238I, "appendonly", d->is_appendonly, d->name.name); } - d->dirty = true; ltfs_set_index_dirty(true, false, vol->index); + ltfs_set_dentry_dirty(d, vol); out_dunlk: _xattr_unlock_dentry(name, true, d, vol); diff --git a/src/libltfs/xml.h b/src/libltfs/xml.h index 5bb967fe..75c22035 100644 --- a/src/libltfs/xml.h +++ b/src/libltfs/xml.h @@ -50,7 +50,7 @@ ** ** Atsushi Abe ** IBM Tokyo Lab., Japan -** piste@jp.ibm.com +** piste@jp.ibm.com ** ************************************************************************************* */ diff --git a/src/libltfs/xml_libltfs.h b/src/libltfs/xml_libltfs.h index 946f662c..0088a6cf 100644 --- a/src/libltfs/xml_libltfs.h +++ b/src/libltfs/xml_libltfs.h @@ -36,7 +36,7 @@ ** ** COMPONENT NAME: IBM Linear Tape File System ** -** FILE NAME: xml.h +** FILE NAME: xml_libltfs.h ** ** DESCRIPTION: Prototypes for XML read/write functions. ** @@ -48,6 +48,10 @@ ** IBM Almaden Research Center ** lucasvr@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ @@ -79,13 +83,13 @@ xmlBufferPtr xml_make_label(const char *creator, tape_partition_t partition, xmlBufferPtr xml_make_schema(const char *creator, const struct ltfs_index *idx); int xml_schema_to_file(const char *filename, const char *creator, const char *reason, const struct ltfs_index *idx); -int xml_schema_to_tape(char *reason, struct ltfs_volume *vol); +int xml_schema_to_tape(char *reason, int type, struct ltfs_volume *vol); /* Functions for reading XML files. See xml_reader_libltfs.c */ int xml_label_from_file(const char *filename, struct ltfs_label *label); int xml_label_from_mem(const char *buf, int buf_size, struct ltfs_label *label); int xml_schema_from_file(const char *filename, struct ltfs_index *idx, struct ltfs_volume *vol); -int xml_schema_from_tape(uint64_t eod_pos, struct ltfs_volume *vol); +int xml_schema_from_tape(uint64_t eod_pos, bool skip_dir, struct ltfs_volume *vol); int xml_extent_symlink_info_from_file(const char *filename, struct dentry *d); #endif /* __xml_libltfs_h */ diff --git a/src/libltfs/xml_reader.c b/src/libltfs/xml_reader.c index c9ddbb4a..b684c2f4 100644 --- a/src/libltfs/xml_reader.c +++ b/src/libltfs/xml_reader.c @@ -48,6 +48,10 @@ ** IBM Almaden Research Center ** lucasvr@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ @@ -541,8 +545,20 @@ int xml_input_tape_read_callback(void *context, char *buffer, int len) /* Try to read a block into the buffer. */ nread = tape_read(ctx->vol->device, ctx->buf, ctx->buf_size, false, - ctx->vol->kmi_handle); + ctx->vol->kmi_handle); ++ctx->current_pos; + + /* Write down the data read to the read cache */ + if (ctx->fd > 0 && nread > 0) { + ret_fd = write(ctx->fd, ctx->buf, nread); + if (ret_fd < 0) { + ltfsmsg(LTFS_ERR, 17244E, (int)errno); + ctx->errno_fd = -LTFS_CACHE_IO; + return -1; + } + } + + /* Condition check of data read */ if (nread < 0) { /* We know we're not at EOD, so read errors are unexpected. */ ltfsmsg(LTFS_ERR, 17039E, (int)nread); diff --git a/src/libltfs/xml_reader_libltfs.c b/src/libltfs/xml_reader_libltfs.c index 6a23e3fa..d604fc37 100644 --- a/src/libltfs/xml_reader_libltfs.c +++ b/src/libltfs/xml_reader_libltfs.c @@ -1490,10 +1490,12 @@ static int _xml_parse_dirtree(xmlTextReaderPtr reader, struct dentry *parent, * with the nodes found during the scanning. * @param reader Source of XML data * @param idx LTFS index + * @param skip_dir skip parsing directory and file * @param vol LTFS volume to which the index belongs. May be NULL. * @return 0 on success or a negative value on error. */ -static int _xml_parse_schema(xmlTextReaderPtr reader, struct ltfs_index *idx, struct ltfs_volume *vol) +static int _xml_parse_schema(xmlTextReaderPtr reader, bool skip_dir, + struct ltfs_index *idx, struct ltfs_volume *vol) { unsigned long long value_int; declare_parser_vars("ltfsindex"); @@ -1589,9 +1591,13 @@ static int _xml_parse_schema(xmlTextReaderPtr reader, struct ltfs_index *idx, st } else if (! strcmp(name, "directory")) { check_required_tag(6); assert_not_empty(); - ret = _xml_parse_dirtree(reader, NULL, idx, vol, NULL); - if (ret < 0) - return ret; + if (skip_dir) { + xml_skip_tag(reader); + } else { + ret = _xml_parse_dirtree(reader, NULL, idx, vol, NULL); + if (ret < 0) + return ret; + } } else if (!strcmp(name, "previousgenerationlocation")) { @@ -1830,7 +1836,7 @@ int xml_label_from_mem(const char *buf, int buf_size, struct ltfs_label *label) * with the nodes found during the scanning. * @param filename XML input file. * @param idx LTFS index. - * @param vol LTFS volume to which the index belongs. May be NULL. + * @param vol LTFS volume to which the index belongs. Can be NULL. * @return 0 on success or a negative value on error. */ int xml_schema_from_file(const char *filename, struct ltfs_index *idx, struct ltfs_volume *vol) @@ -1852,7 +1858,7 @@ int xml_schema_from_file(const char *filename, struct ltfs_index *idx, struct lt * unknown tags modifies the behavior of xmlFreeTextReader so that an additional * xmlDocFree call is required to free all memory. */ doc = xmlTextReaderCurrentDoc(reader); - ret = _xml_parse_schema(reader, idx, vol); + ret = _xml_parse_schema(reader, false, idx, vol); if (ret < 0) ltfsmsg(LTFS_ERR, 17012E, filename, ret); if (doc) @@ -1875,13 +1881,14 @@ int xml_schema_from_file(const char *filename, struct ltfs_index *idx, struct lt * the file mark. * @param eod_pos EOD block position for the current partition, or 0 to assume EOD will not be * encountered during parsing. + * @param skip_dir skip parsing directory * @param vol LTFS volume. * @return 0 on success, 1 if parsing succeeded but no file mark was encountered, * or a negative value on error. */ -int xml_schema_from_tape(uint64_t eod_pos, struct ltfs_volume *vol) +int xml_schema_from_tape(uint64_t eod_pos, bool skip_dir, struct ltfs_volume *vol) { - int ret; + int ret, bk = -1; struct tc_position current_pos; struct xml_input_tape *ctx; xmlParserInputBufferPtr read_buf; @@ -1913,10 +1920,10 @@ int xml_schema_from_tape(uint64_t eod_pos, struct ltfs_volume *vol) ctx->current_pos = current_pos.block; ctx->eod_pos = eod_pos; ctx->saw_small_block = false; - ctx->saw_file_mark = false; - ctx->buf_size = vol->label->blocksize; - ctx->buf_start = 0; - ctx->buf_used = 0; + ctx->saw_file_mark = false; + ctx->buf_size = vol->label->blocksize; + ctx->buf_start = 0; + ctx->buf_used = 0; /* Create input buffer pointer. */ read_buf = xmlParserInputBufferCreateIO(xml_input_tape_read_callback, @@ -1924,6 +1931,8 @@ int xml_schema_from_tape(uint64_t eod_pos, struct ltfs_volume *vol) ctx, XML_CHAR_ENCODING_NONE); if (! read_buf) { ltfsmsg(LTFS_ERR, 17014E); + if (ctx->fd >= 0) + xml_release_file_lock(vol->index_cache_path_r, ctx->fd, bk, false); free(ctx->buf); free(ctx); return -LTFS_LIBXML2_FAILURE; diff --git a/src/libltfs/xml_writer.c b/src/libltfs/xml_writer.c index 1f2944d4..91b3aca5 100644 --- a/src/libltfs/xml_writer.c +++ b/src/libltfs/xml_writer.c @@ -48,6 +48,10 @@ ** IBM Almaden Research Center ** lucasvr@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ diff --git a/src/libltfs/xml_writer_libltfs.c b/src/libltfs/xml_writer_libltfs.c index 7b1ae2be..54ff15b1 100644 --- a/src/libltfs/xml_writer_libltfs.c +++ b/src/libltfs/xml_writer_libltfs.c @@ -60,6 +60,7 @@ #include "fs.h" #include "tape.h" #include "pathname.h" +#include "inc_journal.h" #include "arch/time_internal.h" #ifdef mingw_PLATFORM #include "unicode/umachine.h" @@ -341,10 +342,11 @@ static int _xml_write_file(xmlTextWriterPtr writer, struct dentry *file, struct /* Write dirty file list */ if (sync_list->fp && file->dirty) { fprintf(sync_list->fp, "%s,%"PRIu64"\n", file->name.name, file->size); - file->dirty = false; sync_list->count++; } + file->dirty = false; + return 0; } @@ -398,24 +400,24 @@ static int _xml_write_dirtree(xmlTextWriterPtr writer, struct dentry *dir, HASH_ITER(hh, dir->child_list, list_ptr, list_tmp) { if (list_ptr->d->isdir) { - if (list_ptr->d->vol->index_cache_path && !strcmp(list_ptr->d->name.name, ".LTFSEE_DATA")) { - ret = asprintf(&offset_name, "%s.%s", list_ptr->d->vol->index_cache_path, "offsetcache.new"); + if (list_ptr->d->vol->index_cache_path_w && !strcmp(list_ptr->d->name.name, ".LTFSEE_DATA")) { + ret = asprintf(&offset_name, "%s.%s", list_ptr->d->vol->index_cache_path_w, "offsetcache.new"); if (ret > 0) { arch_fopen(offset_name, "w", offset->fp); free(offset_name); if (!offset->fp) - ltfsmsg(LTFS_WARN, 17248W, "offset cache", list_ptr->d->vol->index_cache_path); + ltfsmsg(LTFS_WARN, 17248W, "offset cache", list_ptr->d->vol->index_cache_path_w); } else - ltfsmsg(LTFS_WARN, 17247W, "offset cache", list_ptr->d->vol->index_cache_path); + ltfsmsg(LTFS_WARN, 17247W, "offset cache", list_ptr->d->vol->index_cache_path_w); - ret = asprintf(&sync_name, "%s.%s", list_ptr->d->vol->index_cache_path, "synclist.new"); + ret = asprintf(&sync_name, "%s.%s", list_ptr->d->vol->index_cache_path_w, "synclist.new"); if (ret > 0) { arch_fopen(sync_name, "w", sync->fp); free(sync_name); if (!sync->fp) - ltfsmsg(LTFS_WARN, 17248W, "sync list", list_ptr->d->vol->index_cache_path); + ltfsmsg(LTFS_WARN, 17248W, "sync list", list_ptr->d->vol->index_cache_path_w); } else - ltfsmsg(LTFS_WARN, 17247W, "sync list", list_ptr->d->vol->index_cache_path); + ltfsmsg(LTFS_WARN, 17247W, "sync list", list_ptr->d->vol->index_cache_path_w); } xml_mktag(_xml_write_dirtree(writer, list_ptr->d, idx, offset, sync), -1); @@ -589,6 +591,433 @@ static int _xml_write_schema(xmlTextWriterPtr writer, const char *creator, return 0; } +#ifdef FORMAT_SPEC25 +static int _xml_write_incremental_dir(xmlTextWriterPtr writer, struct dentry *dir) +{ + /* Handle R/O and timestamp if it is dirty */ + if (dir->dirty) { + xml_mktag(xmlTextWriterWriteElement( + writer, BAD_CAST "readonly", BAD_CAST (dir->readonly ? "true" : "false")), -1); + xml_mktag(_xml_write_dentry_times(writer, dir), -1); + } + + /* Handle UID in any case */ + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST UID_TAGNAME, "%"PRIu64, dir->uid), -1); + + /* Handle extended attribute if it is dirty */ + if (dir->dirty) { + xml_mktag(_xml_write_xattr(writer, dir), -1); + dir->dirty = false; + } + + return 0; +} + +static int _xml_open_incremental_dir_ent(xmlTextWriterPtr writer, struct dentry *dir) +{ + /* Handle R/O and timestamp if it is dirty */ + if (dir->dirty) { + xml_mktag(xmlTextWriterWriteElement( + writer, BAD_CAST "readonly", BAD_CAST (dir->readonly ? "true" : "false")), -1); + xml_mktag(_xml_write_dentry_times(writer, dir), -1); + } + + /* Handle UID in any case */ + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST UID_TAGNAME, "%"PRIu64, dir->uid), -1); + + /* Handle extended attribute if it is dirty */ + if (dir->dirty) { + xml_mktag(_xml_write_xattr(writer, dir), -1); + dir->dirty = false; + } + + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "contents"), -1); + + return 0; +} + +static int _xml_open_incrental_root(xmlTextWriterPtr writer, struct ltfs_volume *vol) +{ + int ret = 0; + struct ltfs_index *idx = vol->index; + struct dentry *dir = idx->root; + + /* Handle name tag */ + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "directory"), -1); + if (idx->volume_name.name) { + xml_mktag(_xml_write_nametype(writer, "name", (struct ltfs_name*)(&idx->volume_name)), -1); + } else { + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "name"), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + } + + ret = _xml_open_incremental_dir_ent(writer, dir); + + return ret; +} + +static inline int _xml_close_incrental_root(xmlTextWriterPtr writer) +{ + xml_mktag(xmlTextWriterEndElement(writer), -1); /* close contents tag */ + xml_mktag(xmlTextWriterEndElement(writer), -1); /* close directory tag */ + return 0; +} + +static int _xml_open_incremental_dir(xmlTextWriterPtr writer, struct dentry *dir) +{ + int ret = 0; + + /* Handle name tag */ + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "directory"), -1); + xml_mktag(_xml_write_nametype(writer, "name", &dir->name), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + + ret = _xml_open_incremental_dir_ent(writer, dir); + + return ret; +} + +static inline int _xml_close_incremental_dir(xmlTextWriterPtr writer) +{ + xml_mktag(xmlTextWriterEndElement(writer), -1); + return 0; +} + +static int _xml_open_incremental_dirs(xmlTextWriterPtr writer, struct incj_path_helper *pm, int offset) +{ + int ret = 0, i = 0; + struct incj_path_element *cur = NULL; + + cur = pm->head; + for (i = 0; i < offset; i++) { + cur = cur->next; + } + + while (cur) { + ret = _xml_open_incremental_dir(writer, cur->d); + if (ret < 0) + break; + cur = cur->next; + } + + return ret; +} + +static int _xml_close_incremental_dirs(xmlTextWriterPtr writer, int pops) +{ + int ret = 0, i = 0; + + for (i = 0; i < pops; i++) { + ret = _xml_close_incremental_dir(writer); + if (ret < 0) + break; + } + + return ret; +} + +static int _xml_write_incremental_delete(xmlTextWriterPtr writer, enum journal_reason reason, struct ltfs_name *name) +{ + /* Open directory tag or file tag based on the provided reason */ + switch (reason) { + case DELETE_DIRECTORY: + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "directory"), -1); + break; + case DELETE_FILE: + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "file"), -1); + break; + default: + ltfsmsg(LTFS_ERR, 17304E, reason); + return -1; + break; + } + + /* Create a name tag */ + xml_mktag(_xml_write_nametype(writer, "name", name), -1); + + /* Create a empty deleted tag */ + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "deleted"), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + + /* Close directory or file tag */ + xml_mktag(xmlTextWriterEndElement(writer), -1); + + return 0; +} + +static int _xml_goto_increment_parent(xmlTextWriterPtr writer, + struct jentry *ent, struct incj_path_helper **cur, + struct ltfs_volume *vol) +{ + int ret = 0, matches = 0, pops = 0; + bool perfect_match = false; + char *parent_path = NULL, *filename = NULL; + struct incj_path_helper *new = NULL; + + parent_path = strdup(ent->id.full_path); + if (!parent_path) { + ltfsmsg(LTFS_ERR, 10001E, "parent path for traveling incremental index dirs"); + return -LTFS_NO_MEMORY; + } + + fs_split_path(parent_path, &filename, strlen(parent_path) + 1); + if (strlen(parent_path) == 0) { + parent_path = "/"; + } + + ret = incj_create_path_helper(parent_path, &new, vol); + if (ret < 0) { + free(parent_path); + return ret; + } + + ret = incj_compare_path(*cur, new, &matches, &pops, &perfect_match); + if (ret < 0) { + incj_destroy_path_helper(new); + free(parent_path); + return ret; + } + + if (pops) { + ret = _xml_close_incremental_dirs(writer, pops); + if (ret < 0) { + incj_destroy_path_helper(new); + free(parent_path); + return ret; + } + } + + if (!perfect_match) + ret = _xml_open_incremental_dirs(writer, new, matches); + + if (!ret) { + incj_destroy_path_helper(*cur); + *cur = new; + } else { + incj_destroy_path_helper(new); + } + + if (strcmp(parent_path, "/")) + free(parent_path); + + return ret; +} + +/** + * Write directory tags for incremental index based on current incremental journal information + * @param writer output pointer + * @param vol ltfs volume + * @param offset_c file pointer to write offest cache + * @param sync_list file pointer to write sync file list + * @return 0 on success or negative on failure + */ +static int _xml_write_inc_journal(xmlTextWriterPtr writer, struct ltfs_volume *vol, + struct ltfsee_cache* offset_c, struct ltfsee_cache* sync_list) +{ + int ret = 0; + bool failed = false; + struct ltfsee_cache offset = {NULL, 0}; /* Cache structure for file offset cache */ + struct ltfsee_cache list = {NULL, 0}; /* Cache structure for sync list */ + struct jentry *ent = NULL, *tmp = NULL; + struct incj_path_helper *cur_parent = NULL; + + /* Create a directory tag for root */ + ret = _xml_open_incrental_root(writer, vol); + if (ret < 0) { + return ret; + } + + ret = incj_create_path_helper("/", &cur_parent, vol); + if (ret < 0) { + return ret; + } + + /* Crawl incremental journal and generate XML tags */ + incj_sort(vol); + HASH_ITER(hh, vol->journal, ent, tmp) { + if (!failed) { + ret = _xml_goto_increment_parent(writer, ent, &cur_parent, vol); + if (!ret) { + switch (ent->reason) { + case CREATE: + /* TODO: Need to support sync cache and offset cache */ + if (ent->dentry->isdir) { + /* Create XML recursively */ + ret = _xml_write_dirtree(writer, ent->dentry, vol->index, &offset, &list); + } else { + /* Create XML for a file */ + ret = _xml_write_file(writer, ent->dentry, &offset, &list); + } + break; + case MODIFY: + if (ent->dentry->isdir) { + /* Create XML for dir (no recursive) */ + ret = _xml_write_incremental_dir(writer, ent->dentry); + } else { + /* Create XML for a file */ + ret = _xml_write_file(writer, ent->dentry, &offset, &list); + } + break; + case DELETE_FILE: + case DELETE_DIRECTORY: + /* Create a delete tag */ + ret = _xml_write_incremental_delete(writer, ent->reason, &ent->name); + break; + default: + ltfsmsg(LTFS_ERR, 17303E, ent->reason); + ret = -LTFS_UNEXPECTED_VALUE; + break; + } + + if (ret < 0) + failed = true; + } else + failed = true; + } + HASH_DEL(vol->journal, ent); + incj_dispose_jentry(ent); + } + + if (cur_parent) { + incj_destroy_path_helper(cur_parent); + } + + /* Clear created directory just in case */ + incj_clear(vol); + + /* Close the directory tag for root */ + ret = _xml_close_incrental_root(writer); + + return ret; +} + +/** + * Generate an XML schema, sending it to a user-provided output (memory or file). + * Note: this function does very little input validation; any user-provided information + * must be verified by the caller. + * @param writer the XML writer to send output to + * @param priv LTFS data + * @param pos position on tape where the schema will be written + * @return 0 on success, negative on failure + */ +static int _xml_write_incremental_schema(xmlTextWriterPtr writer, const char *creator, struct ltfs_volume *vol) +{ + int ret; + size_t i; + char *update_time; + struct ltfsee_cache offset = {NULL, 0}; /* Cache structure for file offset cache */ + struct ltfsee_cache list = {NULL, 0}; /* Cache structure for sync list */ + struct ltfs_index *idx = vol->index; + + ret = xml_format_time(idx->mod_time, &update_time); + if (!update_time) + return -1; + else if (ret == LTFS_TIME_OUT_OF_RANGE) + ltfsmsg(LTFS_WARN, 17224W, "modifytime", (unsigned long long)idx->mod_time.tv_sec); + + ret = xmlTextWriterStartDocument(writer, NULL, "UTF-8", NULL); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 17057E, ret); + return -1; + } + + xmlTextWriterSetIndent(writer, 1); + /* Define INDENT_INDEXES to write Indexes to tape with full indentation. + * This is normally a waste of space, but it may be useful for debugging. */ +#ifdef INDENT_INDEXES + xmlTextWriterSetIndentString(writer, BAD_CAST " "); +#else + xmlTextWriterSetIndentString(writer, BAD_CAST ""); +#endif + + /* write index properties */ + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "ltfsincrementalindex"), -1); + xml_mktag(xmlTextWriterWriteAttribute(writer, BAD_CAST "version", + BAD_CAST LTFS_INDEX_VERSION_STR), -1); + xml_mktag(xmlTextWriterWriteElement(writer, BAD_CAST "creator", BAD_CAST creator), -1); + if (idx->commit_message && strlen(idx->commit_message)) { + xml_mktag(xmlTextWriterWriteFormatElement(writer, BAD_CAST "comment", + "%s", BAD_CAST (idx->commit_message)), -1); + } + xml_mktag(xmlTextWriterWriteElement(writer, BAD_CAST "volumeuuid", BAD_CAST idx->vol_uuid), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "generationnumber", "%u", idx->generation), -1); + xml_mktag(xmlTextWriterWriteElement(writer, BAD_CAST "updatetime", BAD_CAST update_time), -1); + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "location"), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "partition", "%c", idx->selfptr_inc.partition), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "startblock", "%"PRIu64, idx->selfptr_inc.block), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + if (idx->backptr.block) { + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "previousgenerationlocation"), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "partition", "%c", idx->backptr.partition), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "startblock", "%"PRIu64, idx->backptr.block), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + } + + if (idx->backptr_inc.block) { + xml_mktag(xmlTextWriterStartElement(writer, BAD_CAST "previousincrementallocation"), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "partition", "%c", idx->backptr_inc.partition), -1); + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST "startblock", "%"PRIu64, idx->backptr_inc.block), -1); + xml_mktag(xmlTextWriterEndElement(writer), -1); + } + + xml_mktag(xmlTextWriterWriteFormatElement( + writer, BAD_CAST NEXTUID_TAGNAME, "%"PRIu64, idx->uid_number), -1); + + { + char *value = NULL; + + switch (idx->vollock) { + case LOCKED_MAM: + asprintf(&value, "locked"); + break; + case PERMLOCKED_MAM: + asprintf(&value, "permlocked"); + break; + default: + asprintf(&value, "unlocked"); + break; + } + + if (value) + xml_mktag(xmlTextWriterWriteElement(writer, BAD_CAST "volumelockstate", BAD_CAST value), -1); + + free(value); + } + + /* Create XML of update */ + xml_mktag(_xml_write_inc_journal(writer, vol, &offset, &list), -1); + + /* Save unrecognized tags */ + if (idx->tag_count > 0) { + for (i=0; itag_count; ++i) { + if (xmlTextWriterWriteRaw(writer, idx->preserved_tags[i]) < 0) { + ltfsmsg(LTFS_ERR, 17092E, __FUNCTION__); + return -1; + } + } + } + + xml_mktag(xmlTextWriterEndElement(writer), -1); + ret = xmlTextWriterEndDocument(writer); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 17058E, ret); + return -1; + } + + free(update_time); + return 0; +} +#endif + /************************************************************************************** * Global Functions **************************************************************************************/ @@ -810,10 +1239,12 @@ int xml_schema_to_file(const char *filename, const char *creator, /** * Generate an XML Index based on the vol->index->root directory tree. * The generated data are written directly to the tape with the appropriate blocksize. + * @param reason the reason of writing an index on tape + * @param type index type to write (shall be LTFS_FULL_INDEX or LTFS_INCREMENTAL_INDEX) * @param vol LTFS volume. * @return 0 on success or a negative value on error. */ -int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) +int xml_schema_to_tape(char *reason, int type, struct ltfs_volume *vol) { int ret, bk = -1; xmlOutputBufferPtr write_buf; @@ -840,8 +1271,8 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) out_ctx->fd = -1; out_ctx->errno_fd = 0; - if (vol->index_cache_path) - xml_acquire_file_lock(vol->index_cache_path, &out_ctx->fd, &bk, true); + if (vol->index_cache_path_w) + xml_acquire_file_lock(vol->index_cache_path_w, &out_ctx->fd, &bk, true); out_ctx->buf_size = vol->label->blocksize; out_ctx->buf_used = 0; @@ -855,7 +1286,7 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) if (! write_buf) { ltfsmsg(LTFS_ERR, 17053E); if (out_ctx->fd >= 0) - xml_release_file_lock(vol->index_cache_path, out_ctx->fd, bk, false); + xml_release_file_lock(vol->index_cache_path_w, out_ctx->fd, bk, false); free(out_ctx->buf); free(out_ctx); return -LTFS_LIBXML2_FAILURE; @@ -866,7 +1297,7 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) if (! writer) { ltfsmsg(LTFS_ERR, 17054E); if (out_ctx->fd >= 0) - xml_release_file_lock(vol->index_cache_path, out_ctx->fd, bk, false); + xml_release_file_lock(vol->index_cache_path_w, out_ctx->fd, bk, false); xmlOutputBufferClose(write_buf); free(out_ctx->buf); free(out_ctx); @@ -876,7 +1307,19 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) /* Generate the Index. */ asprintf(&creator, "%s - %s", vol->creator, reason); if (creator) { - ret = _xml_write_schema(writer, creator, vol->index); + switch (type) { + case LTFS_FULL_INDEX: + ret = _xml_write_schema(writer, creator, vol->index); + break; +#ifdef FORMAT_SPEC25 + case LTFS_INCREMENTAL_INDEX: + ret = _xml_write_incremental_schema(writer, creator, vol); + break; +#endif + default: + ret = -LTFS_BAD_INDEX_TYPE; + break; + } if (ret < 0) { ltfsmsg(LTFS_ERR, 17055E, ret); } @@ -889,7 +1332,7 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) else if (out_ctx->errno_fd) ret = out_ctx->errno_fd; if (out_ctx->fd >= 0) - xml_release_file_lock(vol->index_cache_path, out_ctx->fd, bk, true); + xml_release_file_lock(vol->index_cache_path_w, out_ctx->fd, bk, true); } else { /* New index is successfully sent to the internal buffer of tape drive */ immed = (strcmp(reason, SYNC_FORMAT) == 0); /* Use immediate write FM only at format */ @@ -899,14 +1342,14 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) * All buffered data, new index and following FM is written on tape correctly. * It's time to unveil the offset cache and sync cache to other programs. */ - if (vol->index_cache_path) - _commit_offset_caches(vol->index_cache_path, vol->index); + if (vol->index_cache_path_w) + _commit_offset_caches(vol->index_cache_path_w, vol->index); } else { ltfsmsg(LTFS_ERR, 11084E, ret); } if (out_ctx->fd >= 0) - xml_release_file_lock(vol->index_cache_path, out_ctx->fd, bk, false); + xml_release_file_lock(vol->index_cache_path_w, out_ctx->fd, bk, false); } /* Update the creator string */ @@ -924,7 +1367,7 @@ int xml_schema_to_tape(char *reason, struct ltfs_volume *vol) } else { ltfsmsg(LTFS_ERR, 10001E, "xml_schema_to_tape: creator string"); xmlFreeTextWriter(writer); - xml_release_file_lock(vol->index_cache_path, out_ctx->fd, bk, true); + xml_release_file_lock(vol->index_cache_path_w, out_ctx->fd, bk, true); ret = -LTFS_NO_MEMORY; } diff --git a/src/ltfs_copyright.h b/src/ltfs_copyright.h index 636ae023..fd9d3795 100644 --- a/src/ltfs_copyright.h +++ b/src/ltfs_copyright.h @@ -3,7 +3,7 @@ ** OO_Copyright_BEGIN ** ** -** Copyright 2010, 2020 IBM Corp. All rights reserved. +** Copyright 2010, 2022 IBM Corp. All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions @@ -44,6 +44,10 @@ ** IBM Almaden Research Center ** bbiskebo@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ diff --git a/src/ltfs_fuse.c b/src/ltfs_fuse.c index db4d3a5c..4e12188f 100644 --- a/src/ltfs_fuse.c +++ b/src/ltfs_fuse.c @@ -3,7 +3,7 @@ ** OO_Copyright_BEGIN ** ** -** Copyright 2010, 2020 IBM Corp. All rights reserved. +** Copyright 2010, 2022 IBM Corp. All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions @@ -54,6 +54,9 @@ ** ************************************************************************************* */ + +#include /* for ULONG_MAX */ + #include "ltfs_fuse.h" #include "libltfs/ltfs_fsops.h" #include "libltfs/iosched.h" @@ -68,7 +71,7 @@ #include "libltfs/arch/win/win_util.h" #endif -#if (__WORDSIZE == 64) +#if (__WORDSIZE == 64 || ULONG_MAX == 0xffffffffffffffffUL) #define FILEHANDLE_TO_STRUCT(fh) ((struct ltfs_file_handle *)(uint64_t)(fh)) #define STRUCT_TO_FILEHANDLE(de) ((uint64_t)(de)) #else @@ -447,8 +450,10 @@ int ltfs_fuse_release(const char *path, struct fuse_file_info *fi) open_write = (((fi->flags & O_WRONLY) == O_WRONLY) || ((fi->flags & O_RDWR) == O_RDWR)); ret = ltfs_fsops_close(file->file_info->dentry_handle, dirty, open_write, true, priv->data); - if (write_index) - ltfs_sync_index(SYNC_CLOSE, true, priv->data); + if (write_index) { + ltfs_set_commit_message_reason(SYNC_CLOSE, priv->data); + ltfs_sync_index(SYNC_CLOSE, true, LTFS_INDEX_AUTO, priv->data); + } _file_close(file->file_info, priv); _free_ltfs_file_handle(file); @@ -1164,11 +1169,9 @@ void ltfs_fuse_umount(void *userdata) if (kmi_initialized(priv->data)) kmi_destroy(priv->data); + ltfs_set_commit_message_reason(SYNC_UNMOUNT, priv->data); ltfs_unmount(SYNC_UNMOUNT, priv->data); - if (priv->capture_index) - ltfs_save_index_to_disk(priv->work_directory, SYNC_UNMOUNT, false, priv->data); - ltfs_request_trace(FUSE_REQ_EXIT(REQ_UNMOUNT), 0, 0); } diff --git a/src/ltfs_fuse.h b/src/ltfs_fuse.h index 3cf79f17..42267bf4 100644 --- a/src/ltfs_fuse.h +++ b/src/ltfs_fuse.h @@ -3,7 +3,7 @@ ** OO_Copyright_BEGIN ** ** -** Copyright 2010, 2020 IBM Corp. All rights reserved. +** Copyright 2010, 2022 IBM Corp. All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions @@ -48,6 +48,10 @@ ** IBM Almaden Research Center ** lucasvr@us.ibm.com ** +** Atsushi Abe +** IBM Tokyo Lab., Japan +** piste@jp.ibm.com +** ************************************************************************************* */ @@ -127,7 +131,7 @@ struct ltfs_fuse_data { unsigned int rollback_gen; /**< Target generation to roll back mount */ int release_device; /**< Release device? */ int allow_other; /**< Allow all users to access the volume? */ - int capture_index; /**< Capture index information to work directory at unmount */ + char *capture_dir; /**< Directory to capture index information */ char *symlink_str; /**< Symbolic Link type fetched by option (live or posix)*/ char *str_append_only_mode; /**< option sting of scsi_append_only_mode */ int append_only_mode; /**< Use append-only mode */ diff --git a/src/main.c b/src/main.c index 533ec70d..dc145016 100644 --- a/src/main.c +++ b/src/main.c @@ -3,7 +3,7 @@ ** OO_Copyright_BEGIN ** ** -** Copyright 2010, 2020 IBM Corp. All rights reserved. +** Copyright 2010, 2022 IBM Corp. All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions @@ -133,7 +133,7 @@ static struct fuse_opt ltfs_options[] = { LTFS_OPT("release_device", release_device, 1), LTFS_OPT("allow_other", allow_other, 1), LTFS_OPT("noallow_other", allow_other, 0), - LTFS_OPT("capture_index", capture_index, 1), + LTFS_OPT("capture_index=%s", capture_dir, 0), LTFS_OPT("symlink_type=%s", symlink_str, 0), LTFS_OPT("scsi_append_only_mode=%s", str_append_only_mode, 0), LTFS_OPT_KEY("-a", KEY_ADVANCED_HELP), @@ -146,35 +146,39 @@ static struct fuse_opt ltfs_options[] = { void single_drive_advanced_usage(const char *default_driver, struct ltfs_fuse_data *priv) { - ltfsresult(14401I); /* LTFS options: */ - ltfsresult(14413I, LTFS_CONFIG_FILE); /* -o config_file= */ - ltfsresult(14404I, LTFS_DEFAULT_WORK_DIR); /* -o work_directory= */ - ltfsresult(14414I); /* -o atime */ - ltfsresult(14440I); /* -o noatime */ - ltfsresult(14415I, default_driver); /* -o tape_backend= */ + ltfsresult(14401I); /* LTFS options: */ + ltfsresult(14403I); /* -o devname= */ + ltfsresult(14413I, LTFS_CONFIG_FILE); /* -o config_file= */ + ltfsresult(14404I, LTFS_DEFAULT_WORK_DIR); /* -o work_directory= */ + ltfsresult(14414I); /* -o atime */ + ltfsresult(14440I); /* -o noatime */ + ltfsresult(14415I, default_driver); /* -o tape_backend= */ ltfsresult(14416I, config_file_get_default_plugin("iosched", priv->config)); /* -o iosched_backend= */ - ltfsresult(14455I, config_file_get_default_plugin("kmi", priv->config)); /* -o kmi_backend= */ - ltfsresult(14417I); /* -o umask= */ - ltfsresult(14418I); /* -o fmask= */ - ltfsresult(14419I); /* -o dmask= */ + ltfsresult(14455I, config_file_get_default_plugin("kmi", priv->config)); /* -o kmi_backend= */ + ltfsresult(14417I); /* -o umask= */ + ltfsresult(14418I); /* -o fmask= */ + ltfsresult(14419I); /* -o dmask= */ ltfsresult(14420I, LTFS_MIN_CACHE_SIZE_DEFAULT); /* -o min_pool_size= */ ltfsresult(14421I, LTFS_MAX_CACHE_SIZE_DEFAULT); /* -o max_pool_size= */ - ltfsresult(14422I); /* -o rules= */ - ltfsresult(14423I); /* -o quiet */ - ltfsresult(14405I); /* -o trace */ - ltfsresult(14467I); /* -o syslogtrace */ - ltfsresult(14424I); /* -o fulltrace */ - ltfsresult(14441I, LTFS_INFO); /* -o verbose= */ - ltfsresult(14425I); /* -o eject */ - ltfsresult(14439I); /* -o noeject */ - ltfsresult(14427I, LONG_MAX / 60); /* -o sync_type=type */ - ltfsresult(14443I); /* -o force_mount_no_eod */ - ltfsresult(14436I); /* -o device_list */ - ltfsresult(14437I); /* -o rollback_mount */ - ltfsresult(14448I); /* -o release_device */ - ltfsresult(14456I); /* -o capture_index */ - ltfsresult(14463I); /* -o scsi_append_only_mode= */ - ltfsresult(14406I); /* -a */ + ltfsresult(14422I); /* -o rules= */ + ltfsresult(14423I); /* -o quiet */ + ltfsresult(14405I); /* -o trace */ + ltfsresult(14467I); /* -o syslogtrace */ + ltfsresult(14424I); /* -o fulltrace */ + ltfsresult(14441I, LTFS_INFO); /* -o verbose= */ + ltfsresult(14425I); /* -o eject */ + ltfsresult(14439I); /* -o noeject */ + ltfsresult(14427I, LONG_MAX / 60); /* -o sync_type=type */ + ltfsresult(14443I); /* -o force_mount_no_eod */ + ltfsresult(14436I); /* -o device_list */ + ltfsresult(14437I); /* -o rollback_mount */ + ltfsresult(14448I); /* -o release_device */ + ltfsresult(14461I); /* -o symlink_type=type */ + ltfsresult(14456I); /* -o capture_index */ + ltfsresult(14463I); /* -o scsi_append_only_mode= */ + ltfsresult(14406I); /* -a */ + ltfsresult(14407I); /* -V, --version */ + ltfsresult(14408I); /* -h, --help */ /* TODO: future use for WORM */ /* set worm rollback flag and rollback_str by this option */ /* ltfsresult(14468I); */ /* -o rollback_mount_no_eod */ @@ -194,30 +198,30 @@ void usage(char *progname, struct ltfs_fuse_data *priv) if (ret == 0) default_device = ltfs_default_device_name(priv->tape_plugin.ops); - ltfsresult(14400I, progname); /* usage: %s mountpoint [options] */ + ltfsresult(14400I, progname); /* usage: %s mountpoint [options] */ fprintf(stderr, "\n"); - ltfsresult(14401I); /* LTFS options: */ + ltfsresult(14401I); /* LTFS options: */ if (default_device) - ltfsresult(14402I, default_device); /* -o devname= */ + ltfsresult(14402I, default_device); /* -o devname= */ else - ltfsresult(14403I); /* -o devname= */ - ltfsresult(14404I, LTFS_DEFAULT_WORK_DIR); /* -o work_directory= */ - ltfsresult(14405I); /* -o trace */ - ltfsresult(14425I); /* -o eject */ - ltfsresult(14427I, LONG_MAX / 60); /* -o sync_type=type */ - ltfsresult(14443I); /* -o force_mount_no_eod */ - ltfsresult(14436I); /* -o device_list */ - ltfsresult(14437I); /* -o rollback_mount */ - ltfsresult(14448I); /* -o release_device */ - ltfsresult(14461I); /* -o symlink_type=type */ - ltfsresult(14406I); /* -a */ - ltfsresult(14407I); /* -V, --version */ - ltfsresult(14408I); /* -h, --help */ + ltfsresult(14403I); /* -o devname= */ + ltfsresult(14404I, LTFS_DEFAULT_WORK_DIR); /* -o work_directory= */ + ltfsresult(14405I); /* -o trace */ + ltfsresult(14425I); /* -o eject */ + ltfsresult(14427I, LONG_MAX / 60); /* -o sync_type=type */ + ltfsresult(14443I); /* -o force_mount_no_eod */ + ltfsresult(14436I); /* -o device_list */ + ltfsresult(14437I); /* -o rollback_mount */ + ltfsresult(14448I); /* -o release_device */ + ltfsresult(14461I); /* -o symlink_type=type */ + ltfsresult(14406I); /* -a */ + ltfsresult(14407I); /* -V, --version */ + ltfsresult(14408I); /* -h, --help */ fprintf(stderr, "\n"); - ltfsresult(14409I); /* FUSE options: */ - ltfsresult(14410I); /* -o umask=M */ - ltfsresult(14411I); /* -o uid=N */ - ltfsresult(14412I); /* -o gid=N */ + ltfsresult(14409I); /* FUSE options: */ + ltfsresult(14410I); /* -o umask=M */ + ltfsresult(14411I); /* -o uid=N */ + ltfsresult(14412I); /* -o gid=N */ fprintf(stderr, "\n"); fprintf(stderr, "\n"); @@ -704,7 +708,7 @@ int main(int argc, char **argv) if (priv->device_list) { ret = show_device_list(priv); ltfs_finish(); - return (ret != 0) ? 0 : 1; + return ret ? 1 : 0; } /* Validate sync option */ @@ -877,11 +881,14 @@ int main(int argc, char **argv) /* Make sure we have a device name */ if (! priv->devname) { - priv->devname = ltfs_default_device_name(priv->tape_plugin.ops); - if (! priv->devname) { - /* The backend \'%s\' does not have a default device */ - ltfsmsg(LTFS_ERR, 14009E, priv->tape_backend_name); - return 1; + /* Accept no devname when accessible index file is specified by '-o rollback_mount' */ + if ( !priv->rollback_str || access(priv->rollback_str, R_OK) ) { + priv->devname = ltfs_default_device_name(priv->tape_plugin.ops); + if (! priv->devname) { + /* The backend \'%s\' does not have a default device */ + ltfsmsg(LTFS_ERR, 14009E, priv->tape_backend_name); + return 1; + } } } @@ -922,9 +929,10 @@ int main(int argc, char **argv) int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) { - int ret, altret; + int ret, altret, fd = -1; char *index_rules_utf8; - char fsname[strlen(priv->devname) + 16]; + char *fsname_base = "-ofsname=ltfs:"; + char *fsname; char *invalid_start; #ifdef __APPLE__ char *opt_volname = NULL; @@ -934,6 +942,18 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) int i; bool is_worm = false, is_ro = false; + if (priv->devname) { + fsname = calloc(1, strlen(fsname_base) + strlen(priv->devname) + 1); + } else if (priv->rollback_str) { + fsname = calloc(1, strlen(fsname_base) + strlen(priv->rollback_str) + 1); + } else { + fsname = calloc(1, strlen(fsname_base) + 1); + } + if (!fsname) { + /* Memory allocation failed */ + ltfsmsg(LTFS_ERR, 10001E, "fsname"); + return -ENOMEM; + } /* Setup signal handler to terminate cleanly */ ret = ltfs_set_signal_handlers(); if (ret < 0) { @@ -943,11 +963,13 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) /* Validate rollback_mount option */ if (priv->rollback_str) { - errno = 0; - priv->rollback_gen = strtoul(priv->rollback_str, &invalid_start, 0); - if( (*invalid_start != '\0') || priv->rollback_gen == 0 ) { - ltfsmsg(LTFS_ERR, 14091E, priv->rollback_str); - return 1; + if (access(priv->rollback_str, R_OK)) { + errno = 0; + priv->rollback_gen = strtoul(priv->rollback_str, &invalid_start, 0); + if( (*invalid_start != '\0') || priv->rollback_gen == 0 ) { + ltfsmsg(LTFS_ERR, 14091E, priv->rollback_str); + return 1; + } } } @@ -983,7 +1005,12 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) } /* Set file system name to "ltfs:devname" in case FUSE doesn't pick it up */ - snprintf(fsname, sizeof(fsname), "-ofsname=ltfs:%s", priv->devname); + strcpy(fsname, fsname_base); + if (priv->devname) { + strcat(fsname, priv->devname); + } else if (priv->rollback_str) { + strcat(fsname, priv->rollback_str); + } ret = fuse_opt_add_arg(args, fsname); if (ret < 0) { /* Could not enable FUSE option */ @@ -1000,145 +1027,163 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) ltfs_use_atime(priv->atime, priv->data); ltfs_set_work_dir(priv->work_directory, priv->data); - if (ltfs_device_open(priv->devname, priv->tape_plugin.ops, priv->data) < 0) { - /* Could not open device */ - ltfsmsg(LTFS_ERR, 10004E, priv->devname); - ltfs_volume_free(&priv->data); - return 1; - } - - if (priv->release_device) { - ltfs_release_medium(priv->data); - ltfs_device_close(priv->data); - ltfs_volume_free(&priv->data); - return 0; - } - - /* Parse backend options */ - if (ltfs_parse_tape_backend_opts(args, priv->data)) { - /* Backend option parsing failed */ - ltfsmsg(LTFS_ERR, 14012E); - ltfs_volume_free(&priv->data); - return 1; - } - - if (priv->kmi_backend_name) { - if (kmi_init(&priv->kmi_plugin, priv->data) < 0) { - /* Encryption function disabled. */ - ltfsmsg(LTFS_ERR, 14089E); + if (priv->devname) { + if (ltfs_device_open(priv->devname, priv->tape_plugin.ops, priv->data) < 0) { + /* Could not open device */ + ltfsmsg(LTFS_ERR, 10004E, priv->devname); ltfs_volume_free(&priv->data); return 1; } - if (ltfs_parse_kmi_backend_opts(args, priv->data)) { + if (priv->release_device) { + ltfs_release_medium(priv->data); + ltfs_device_close(priv->data); + ltfs_volume_free(&priv->data); + return 0; + } + + /* Parse backend options */ + if (ltfs_parse_tape_backend_opts(args, priv->data)) { /* Backend option parsing failed */ - ltfsmsg(LTFS_ERR, 14090E); + ltfsmsg(LTFS_ERR, 14012E); ltfs_volume_free(&priv->data); return 1; } - if (tape_clear_key(priv->data->device, priv->data->kmi_handle) < 0) - return 1; - } + if (priv->kmi_backend_name) { + if (kmi_init(&priv->kmi_plugin, priv->data) < 0) { + /* Encryption function disabled. */ + ltfsmsg(LTFS_ERR, 14089E); + ltfs_volume_free(&priv->data); + return 1; + } - /* Setup tape drive */ - ltfs_load_tape(priv->data); - ret = ltfs_wait_device_ready(priv->data); - if (ret < 0) { - ltfsmsg(LTFS_ERR, 14075E); - ltfs_volume_free(&priv->data); - return 1; - } + if (ltfs_parse_kmi_backend_opts(args, priv->data)) { + /* Backend option parsing failed */ + ltfsmsg(LTFS_ERR, 14090E); + ltfs_volume_free(&priv->data); + return 1; + } - priv->data->append_only_mode = (bool)priv->append_only_mode; - if (ltfs_setup_device(priv->data)) { - ltfsmsg(LTFS_ERR, 14075E); - ltfs_volume_free(&priv->data); - return 1; - } + if (tape_clear_key(priv->data->device, priv->data->kmi_handle) < 0) + return 1; + } - /* Check EOD validation is skipped or not */ - if (priv->skip_eod_check) { - ltfsmsg(LTFS_INFO, 14076I); - ltfsmsg(LTFS_INFO, 14077I); - ltfs_set_eod_check(! priv->skip_eod_check, priv->data); - } + /* Setup tape drive */ + ltfs_load_tape(priv->data); + ret = ltfs_wait_device_ready(priv->data); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 14075E); + ltfs_volume_free(&priv->data); + return 1; + } - /* Validate symbolic link type */ - priv->data->livelink = false; - if (priv->symlink_str) { - if (strcasecmp(priv->symlink_str, "live") == 0) - priv->data->livelink = true; - else if (strcasecmp(priv->symlink_str, "posix") == 0) - priv->data->livelink = false; - else { - ltfsmsg(LTFS_ERR, 14093E, priv->symlink_str); + priv->data->append_only_mode = (bool)priv->append_only_mode; + if (ltfs_setup_device(priv->data)) { + ltfsmsg(LTFS_ERR, 14075E); + ltfs_volume_free(&priv->data); return 1; } - ltfsmsg(LTFS_INFO, 14092I, priv->symlink_str); - } - /* Mount the volume */ - ltfs_set_traverse_mode(TRAVERSE_BACKWARD, priv->data); - if (ltfs_mount(false, false, false, false, priv->rollback_gen, priv->data) < 0) { - ltfsmsg(LTFS_ERR, 14013E); - ltfs_volume_free(&priv->data); - return 1; - } + /* Check EOD validation is skipped or not */ + if (priv->skip_eod_check) { + ltfsmsg(LTFS_INFO, 14076I); + ltfsmsg(LTFS_INFO, 14077I); + ltfs_set_eod_check(! priv->skip_eod_check, priv->data); + } - ret = tape_get_worm_status(priv->data->device, &is_worm); - if (ret != 0 || is_worm) { - ltfsmsg(LTFS_ERR, 14116E, ret); - ltfs_volume_free(&priv->data); - return 1; - } + /* Validate symbolic link type */ + priv->data->livelink = false; + if (priv->symlink_str) { + if (strcasecmp(priv->symlink_str, "live") == 0) + priv->data->livelink = true; + else if (strcasecmp(priv->symlink_str, "posix") == 0) + priv->data->livelink = false; + else { + ltfsmsg(LTFS_ERR, 14093E, priv->symlink_str); + return 1; + } + ltfsmsg(LTFS_INFO, 14092I, priv->symlink_str); + } - /* Set up index criteria */ - if (priv->index_rules) { - ret = pathname_format(priv->index_rules, &index_rules_utf8, false, false); - if (ret < 0) { - /* Could not format data placement rules. */ - ltfsmsg(LTFS_ERR, 14016E, ret); - ltfs_volume_free(&priv->data); - return 1; + /* Mount the volume */ + ltfs_set_traverse_mode(TRAVERSE_BACKWARD, priv->data); + if (priv->rollback_str) { + if (ltfs_mount_indexfile(priv->rollback_str, true, priv->data) < 0) { + ltfsmsg(LTFS_ERR, 14013E, "index file"); + ltfs_volume_free(&priv->data); + return 1; + } + } else { + if (ltfs_mount(false, false, false, false, priv->rollback_gen, priv->data) < 0) { + ltfsmsg(LTFS_ERR, 14013E, "device"); + ltfs_volume_free(&priv->data); + return 1; + } } - ret = ltfs_override_policy(index_rules_utf8, false, priv->data); - free(index_rules_utf8); - if (ret == -LTFS_POLICY_IMMUTABLE) { - /* Volume doesn't allow override. Ignoring user-specified criteria. */ - ltfsmsg(LTFS_WARN, 14015W); - } else if (ret < 0) { - /* Could not parse data placement rules */ - ltfsmsg(LTFS_ERR, 14017E, ret); + + ret = tape_get_worm_status(priv->data->device, &is_worm); + if (ret != 0 || is_worm) { + ltfsmsg(LTFS_ERR, 14116E, ret); ltfs_volume_free(&priv->data); return 1; } - } - /* Configure I/O scheduler cache */ - ltfs_set_scheduler_cache(priv->min_pool_size, priv->max_pool_size, priv->data); - - /* mount read-only if underlying medium is write-protected */ - ret = ltfs_get_tape_readonly(priv->data); - switch (ret) { - case 0: - case -LTFS_WRITE_PROTECT: - case -LTFS_WRITE_ERROR: - case -LTFS_NO_SPACE: - case -LTFS_LESS_SPACE: - case -LTFS_RDONLY_DEN_DRV: - /* Do nothing */ - break; - default: - /* Fail immidiatly when return code is NOT success or NOT possible R/O related errors */ - /* Could not get read-only status of medium */ - ltfsmsg(LTFS_ERR, 14018E); + /* Set up index criteria */ + if (priv->index_rules) { + ret = pathname_format(priv->index_rules, &index_rules_utf8, false, false); + if (ret < 0) { + /* Could not format data placement rules. */ + ltfsmsg(LTFS_ERR, 14016E, ret); + ltfs_volume_free(&priv->data); + return 1; + } + ret = ltfs_override_policy(index_rules_utf8, false, priv->data); + free(index_rules_utf8); + if (ret == -LTFS_POLICY_IMMUTABLE) { + /* Volume doesn't allow override. Ignoring user-specified criteria. */ + ltfsmsg(LTFS_WARN, 14015W); + } else if (ret < 0) { + /* Could not parse data placement rules */ + ltfsmsg(LTFS_ERR, 14017E, ret); + ltfs_volume_free(&priv->data); + return 1; + } + } + + /* Configure I/O scheduler cache */ + ltfs_set_scheduler_cache(priv->min_pool_size, priv->max_pool_size, priv->data); + + /* mount read-only if underlying medium is write-protected */ + ret = ltfs_get_tape_readonly(priv->data); + switch (ret) { + case 0: + case -LTFS_WRITE_PROTECT: + case -LTFS_WRITE_ERROR: + case -LTFS_NO_SPACE: + case -LTFS_LESS_SPACE: + case -LTFS_RDONLY_DEN_DRV: + /* Do nothing */ + break; + default: + /* Fail immidiatly when return code is NOT success or NOT possible R/O related errors */ + /* Could not get read-only status of medium */ + ltfsmsg(LTFS_ERR, 14018E); + ltfs_volume_free(&priv->data); + return 1; + break; + } + } else { + /* try to mount from index file (meta-only mount) */ + if (ltfs_mount_indexfile(priv->rollback_str, false, priv->data) < 0) { + ltfsmsg(LTFS_ERR, 14013E, "index file"); ltfs_volume_free(&priv->data); return 1; - break; + } + ret = 0; } - if (ret < 0 || priv->rollback_gen != 0) { + if (ret < 0 || priv->rollback_gen || priv->rollback_str) { switch (ret) { case -LTFS_WRITE_PROTECT: case -LTFS_WRITE_ERROR: @@ -1171,9 +1216,25 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) is_ro = true; break; default: - /* Rollback mount is specified */ - ltfsmsg(LTFS_INFO, 14072I, priv->rollback_gen); - is_ro = true; + if (!ret && priv->rollback_gen) { + /* Rollback mount is specified */ + ltfsmsg(LTFS_INFO, 14072I, priv->rollback_gen); + is_ro = true; + } else if (!ret && priv->rollback_str) { + if (priv->devname) { + /* Rollback mount (index mount) is specified */ + ltfsmsg(LTFS_INFO, 14119I, priv->rollback_str); + } else { + /* Rollback mount (meta-only mount) is specified */ + ltfsmsg(LTFS_INFO, 14117I, priv->rollback_str); + } + is_ro = true; + } else { + /* Unexpected condition */ + ltfsmsg(LTFS_ERR, 14118E, ret); + ltfs_volume_free(&priv->data); + return 1; + } break; } @@ -1188,10 +1249,42 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) } } + /* Configure index capturing */ + if (priv->capture_dir) { + if (HAVE_BARCODE(priv->data)) + ret = asprintf(&priv->data->index_cache_path_w, "%s/%s.schema", + priv->capture_dir, priv->data->label->barcode); + else + ret = asprintf(&priv->data->index_cache_path_w, "%s/%s.schema", + priv->capture_dir, priv->data->label->vol_uuid); + + if (ret > 0) { + fd = open(priv->data->index_cache_path_w, O_WRONLY | O_BINARY | O_CREAT, + S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP); + if (fd < 0) { + ltfsmsg(LTFS_WARN, 14120W, priv->capture_dir, errno); + free(priv->data->index_cache_path_w); + priv->data->index_cache_path_w = NULL; + } else { + ltfsmsg(LTFS_INFO, 14121I, priv->data->index_cache_path_w); + close(fd); + fd = -1; + } + } else { + ltfsmsg(LTFS_ERR, 10001E, "capture_dir"); + ltfs_volume_free(&priv->data); + return 1; + } + } else { + ltfsmsg(LTFS_INFO, 14122I); + ret = 0; + } + /* Cleanup signal handler */ ret = ltfs_unset_signal_handlers(); if (ret < 0) { ltfsmsg(LTFS_ERR, 10014E); + ltfs_volume_free(&priv->data); return 1; } @@ -1257,7 +1350,7 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) return 1; } - if (priv->eject) + if (priv->devname && priv->eject) ltfs_eject_tape(false, priv->data); /* close the volume */ @@ -1265,7 +1358,8 @@ int single_drive_main(struct fuse_args *args, struct ltfs_fuse_data *priv) if (opt_volname) free(opt_volname); #endif - ltfs_device_close(priv->data); + if (priv->devname) + ltfs_device_close(priv->data); ltfs_volume_free(&priv->data); ltfs_unset_signal_handlers(); diff --git a/src/tape_drivers/freebsd/cam/cam_tc.c b/src/tape_drivers/freebsd/cam/cam_tc.c index 377af2ed..64241ab8 100644 --- a/src/tape_drivers/freebsd/cam/cam_tc.c +++ b/src/tape_drivers/freebsd/cam/cam_tc.c @@ -282,10 +282,9 @@ int camtape_open(const char *devname, void **handle) softc->vendor = get_vendor_id(vendor); struct supported_device **cur = get_supported_devs(softc->vendor); while(*cur) { - if ((! strncmp((char*)softc->cd->inq_data.vendor, (*cur)->vendor_id, - strlen((*cur)->vendor_id)) ) && - (! strncmp((char*)softc->cd->inq_data.product, (*cur)->product_id, - strlen((*cur)->product_id)) ) ) { + if ((! strncmp((char*)softc->cd->inq_data.vendor, (*cur)->vendor_id, strlen((*cur)->vendor_id)) ) && + (! strncmp((char*)softc->cd->inq_data.product, (*cur)->product_id, strlen((*cur)->product_id)) ) ) { + softc->vendor = (*cur)->vendor_type; drive_type = (*cur)->drive_type; break; } @@ -1133,7 +1132,6 @@ int camtape_load(void *device, struct tc_position *pos) softc->force_readperm = DEFAULT_READPERM; softc->write_counter = 0; softc->read_counter = 0; - softc->cart_type = buf[2]; softc->density_code = buf[8]; if (softc->vendor == VENDOR_HP) { @@ -1781,7 +1779,10 @@ int camtape_remaining_capacity(void *device, struct tc_remaining_cap *cap) int rc; ltfs_profiler_add_entry(softc->profiler, NULL, TAPEBEND_REQ_ENTER(REQ_TC_REMAINCAP)); - if (IS_LTO(softc->drive_type) && (DRIVE_GEN(softc->drive_type) == 0x05)) { + if ((IS_LTO(softc->drive_type) && (DRIVE_GEN(softc->drive_type) == 0x05)) || + (softc->vendor == VENDOR_HP && IS_LTO(softc->drive_type) && (DRIVE_GEN(softc->drive_type) == 0x06)) || + (softc->vendor == VENDOR_QUANTUM_B && IS_LTO(softc->drive_type))) { + /* Issue LogPage 0x31 */ rc = camtape_logsense(device, LOG_TAPECAPACITY, (uint8_t)0, logdata, LOGSENSEPAGE); if (rc < 0) { @@ -1984,7 +1985,7 @@ int camtape_modeselect(void *device, unsigned char *buf, const size_t size) /*retries*/ 1, /*cbfcnp*/ NULL, /*tag_action*/ MSG_SIMPLE_Q_TAG, - /*scsi_page_fmt*/ 0, + /*scsi_page_fmt*/ 1, /*save_pages*/ 0, /*param_buf*/ buf, /*param_len*/ size, diff --git a/src/tape_drivers/generic/file/filedebug_tc.c b/src/tape_drivers/generic/file/filedebug_tc.c index 615517f0..071e87c0 100644 --- a/src/tape_drivers/generic/file/filedebug_tc.c +++ b/src/tape_drivers/generic/file/filedebug_tc.c @@ -1513,6 +1513,18 @@ static inline int _sanitize_tape(struct filedebug_data *state) ret = -EDEV_MEDIUM_FORMAT_ERROR; break; } + } else if (gen == DRIVE_GEN_JAG7) { + switch (state->conf.cart_type) { + case TC_MP_JF: + state->is_worm = false; + break; + default: + ltfsmsg(LTFS_INFO, 30086I, "TS1170", state->conf.cart_type); + state->is_worm = false; + state->unsupported_tape = true; + ret = -EDEV_MEDIUM_FORMAT_ERROR; + break; + } } else { ltfsmsg(LTFS_INFO, 30086I, "Unexpected Drive", state->conf.cart_type); state->is_worm = false; diff --git a/src/tape_drivers/hp_tape.c b/src/tape_drivers/hp_tape.c index 35fd6002..bbedb8a8 100644 --- a/src/tape_drivers/hp_tape.c +++ b/src/tape_drivers/hp_tape.c @@ -63,10 +63,12 @@ #include "libltfs/ltfs_endian.h" struct supported_device *hp_supported_drives[] = { - TAPEDRIVE( HP_VENDOR_ID, "Ultrium 5-SCSI", DRIVE_LTO5, "[Ultrium 5-SCSI]" ), /* HP Ultrium Gen 5 */ - TAPEDRIVE( HP_VENDOR_ID, "Ultrium 6-SCSI", DRIVE_LTO6, "[Ultrium 6-SCSI]" ), /* HP Ultrium Gen 6 */ - TAPEDRIVE( HP_VENDOR_ID, "Ultrium 7-SCSI", DRIVE_LTO7, "[Ultrium 7-SCSI]" ), /* HP Ultrium Gen 7 */ - TAPEDRIVE( HPE_VENDOR_ID, "Ultrium 8-SCSI", DRIVE_LTO8, "[Ultrium 8-SCSI]" ), /* HPE Ultrium Gen 8 */ + TAPEDRIVE( HP_VENDOR_ID, "Ultrium 5-SCSI", VENDOR_HP, DRIVE_LTO5, "[Ultrium 5-SCSI]" ), /* HP Ultrium Gen 5 */ + TAPEDRIVE( HP_VENDOR_ID, "Ultrium 6-SCSI", VENDOR_HP, DRIVE_LTO6, "[Ultrium 6-SCSI]" ), /* HP Ultrium Gen 6 */ + TAPEDRIVE( HP_VENDOR_ID, "Ultrium 7-SCSI", VENDOR_HP, DRIVE_LTO7, "[Ultrium 7-SCSI]" ), /* HP Ultrium Gen 7 */ + TAPEDRIVE( HPE_VENDOR_ID, "Ultrium 8-SCSI", VENDOR_HP, DRIVE_LTO8, "[Ultrium 8-SCSI]" ), /* HPE Ultrium Gen 8 */ + TAPEDRIVE( TANDBERG_VENDOR_ID, "LTO-5 HH", VENDOR_HP, DRIVE_LTO5_HH, "[TANDBERG LTO5]" ), /* TANDBERG LTO-5 HH */ + TAPEDRIVE( TANDBERG_VENDOR_ID, "LTO-6 HH", VENDOR_HP, DRIVE_LTO6_HH, "[TANDBERG LTO6]" ), /* TANDBERG LTO-6 HH */ /* End of supported_devices */ NULL }; diff --git a/src/tape_drivers/hp_tape.h b/src/tape_drivers/hp_tape.h index 5d60dd9a..5d4a38a5 100644 --- a/src/tape_drivers/hp_tape.h +++ b/src/tape_drivers/hp_tape.h @@ -65,8 +65,9 @@ extern "C" { #endif -#define HP_VENDOR_ID "HP" -#define HPE_VENDOR_ID "HPE" +#define HP_VENDOR_ID "HP" +#define HPE_VENDOR_ID "HPE" +#define TANDBERG_VENDOR_ID "TANDBERG" extern struct error_table hp_tape_errors[]; diff --git a/src/tape_drivers/linux/lin_tape/lin_tape_ibmtape.c b/src/tape_drivers/linux/lin_tape/lin_tape_ibmtape.c index f8c1448c..d70af0fa 100644 --- a/src/tape_drivers/linux/lin_tape/lin_tape_ibmtape.c +++ b/src/tape_drivers/linux/lin_tape/lin_tape_ibmtape.c @@ -2205,7 +2205,7 @@ int lin_tape_ibmtape_format(void *device, TC_FORMAT_TYPE format, const char *vol * @param page page code of log sense * @param buf pointer to buffer to store log data * @param size length of the buffer - * @return 0 on success or a negative value on error + * @return Page length on success or a negative value on error. */ #define MAX_UINT16 (0x0000FFFF) diff --git a/src/tape_drivers/linux/sg/sg_tape.c b/src/tape_drivers/linux/sg/sg_tape.c index 19f2beb8..86d93e95 100644 --- a/src/tape_drivers/linux/sg/sg_tape.c +++ b/src/tape_drivers/linux/sg/sg_tape.c @@ -497,12 +497,12 @@ static int _raw_open(struct sg_data *priv) priv->dev.fd = -1; return ret; } - priv->vendor = get_vendor_id(id_data.vendor_id); - struct supported_device **cur = get_supported_devs(priv->vendor); + struct supported_device **cur = get_supported_devs(id_data.vendor_id); while(*cur) { if((! strncmp(id_data.vendor_id, (*cur)->vendor_id, strlen((*cur)->vendor_id)) ) && (! strncmp(id_data.product_id, (*cur)->product_id, strlen((*cur)->product_id)) ) ) { + priv->vendor = (*cur)->vendor_type; drive_type = (*cur)->drive_type; break; } @@ -2753,11 +2753,17 @@ int sg_load(void *device, struct tc_position *pos) priv->density_code = buf[8]; - if (priv->vendor == VENDOR_HP) { + if (buf[2] == 0x00 || buf[2] == 0x01) { + /* + * Non-IBM drive doesn't have cartridge type so need to assume from density code. + */ priv->cart_type = assume_cart_type(priv->density_code); if (buf[2] == 0x01) priv->is_worm = true; } else { + /* + * IBM drive haves cartridge type in buf[2] like TC_MP_LTO5D_CART. + */ priv->cart_type = buf[2]; } @@ -3031,7 +3037,10 @@ int sg_remaining_capacity(void *device, struct tc_remaining_cap *cap) memset(buffer, 0, LOGSENSEPAGE); - if (IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x05)) { + if ((IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x05)) || + (priv->vendor == VENDOR_HP && IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x06)) || + (priv->vendor == VENDOR_QUANTUM_B && IS_LTO(priv->drive_type))) { + /* Use LogPage 0x31 */ ret = sg_logsense(device, (uint8_t)LOG_TAPECAPACITY, (uint8_t)0, (void *)buffer, LOGSENSEPAGE); if(ret < 0) diff --git a/src/tape_drivers/osx/iokit/iokit_tape.c b/src/tape_drivers/osx/iokit/iokit_tape.c index 76f14628..680615ab 100644 --- a/src/tape_drivers/osx/iokit/iokit_tape.c +++ b/src/tape_drivers/osx/iokit/iokit_tape.c @@ -856,14 +856,12 @@ int iokit_open(const char *devname, void **handle) } strncpy(priv->drive_serial, id_data.unit_serial, UNIT_SERIAL_LENGTH - 1); - /* Convert vendor_id in inq buffer to integer based id */ - priv->vendor = get_vendor_id(id_data.vendor_id); - /* Check the drive is supportable */ - struct supported_device **cur = get_supported_devs(priv->vendor); + struct supported_device **cur = get_supported_devs(id_data.vendor_id); while(cur && *cur) { if((! strncmp(id_data.vendor_id, (*cur)->vendor_id, strlen((*cur)->vendor_id)) ) && (! strncmp(id_data.product_id, (*cur)->product_id, strlen((*cur)->product_id)) ) ) { + priv->vendor = (*cur)->vendor_type; drive_type = (*cur)->drive_type; break; } @@ -2066,14 +2064,26 @@ int iokit_load(void *device, struct tc_position *pos) priv->density_code = buf[8]; - if (priv->vendor == VENDOR_HP) { + if (buf[2] == 0x00 || buf[2] == 0x01) { + /* + * Non-IBM drive doesn't have cartridge type so need to assume from density code. + */ priv->cart_type = assume_cart_type(priv->density_code); if (buf[2] == 0x01) priv->is_worm = true; } else { + /* + * IBM drive haves cartridge type in buf[2] like TC_MP_LTO5D_CART. + */ priv->cart_type = buf[2]; } + if (priv->cart_type == 0x00) { + ltfsmsg(LTFS_WARN, 30871W); + ltfs_profiler_add_entry(priv->profiler, NULL, TAPEBEND_REQ_EXIT(REQ_TC_LOAD)); + return 0; + } + ret = is_supported_tape(priv->cart_type, priv->density_code, &(priv->is_worm)); if(ret == -LTFS_UNSUPPORTED_MEDIUM) ltfsmsg(LTFS_INFO, 30831I, priv->cart_type, priv->density_code); @@ -2302,7 +2312,10 @@ int iokit_remaining_capacity(void *device, struct tc_remaining_cap *cap) memset(&buffer, 0, LOGSENSEPAGE); - if (IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x05)) { + if ((IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x05)) || + (priv->vendor == VENDOR_HP && IS_LTO(priv->drive_type) && (DRIVE_GEN(priv->drive_type) == 0x06)) || + (priv->vendor == VENDOR_QUANTUM_B && IS_LTO(priv->drive_type))) { + /* Use LogPage 0x31 */ ret = iokit_logsense(device, (uint8_t)LOG_TAPECAPACITY, (uint8_t)0, (void *)buffer, LOGSENSEPAGE); if(ret < 0) diff --git a/src/tape_drivers/quantum_tape.c b/src/tape_drivers/quantum_tape.c index a5c477dd..f3a71a0a 100644 --- a/src/tape_drivers/quantum_tape.c +++ b/src/tape_drivers/quantum_tape.c @@ -63,10 +63,12 @@ #include "libltfs/ltfs_endian.h" struct supported_device *quantum_supported_drives[] = { - TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH5", DRIVE_LTO5_HH, "[ULTRIUM-HH5]" ), /* QUANTUM Ultrium Gen 5 Half-High */ - TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH6", DRIVE_LTO6_HH, "[ULTRIUM-HH6]" ), /* QUANTUM Ultrium Gen 6 Half-High */ - TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH7", DRIVE_LTO7_HH, "[ULTRIUM-HH7]" ), /* QUANTUM Ultrium Gen 7 Half-High */ - TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH8", DRIVE_LTO8_HH, "[ULTRIUM-HH8]" ), /* QUANTUM Ultrium Gen 8 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH5", VENDOR_QUANTUM, DRIVE_LTO5_HH, "[ULTRIUM-HH5]" ), /* QUANTUM Ultrium Gen 5 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH6", VENDOR_QUANTUM, DRIVE_LTO6_HH, "[ULTRIUM-HH6]" ), /* QUANTUM Ultrium Gen 6 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH7", VENDOR_QUANTUM, DRIVE_LTO7_HH, "[ULTRIUM-HH7]" ), /* QUANTUM Ultrium Gen 7 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM-HH8", VENDOR_QUANTUM, DRIVE_LTO8_HH, "[ULTRIUM-HH8]" ), /* QUANTUM Ultrium Gen 8 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM 5", VENDOR_QUANTUM_B, DRIVE_LTO5_HH, "[ULTRIUM-5]" ), /* Another QUANTUM Ultrium Gen 5 Half-High */ + TAPEDRIVE( QUANTUM_VENDOR_ID, "ULTRIUM 6", VENDOR_QUANTUM_B, DRIVE_LTO6_HH, "[ULTRIUM-6]" ), /* Another QUANTUM Ultrium Gen 6 Half-High */ /* End of supported_devices */ NULL }; diff --git a/src/tape_drivers/tape_drivers.h b/src/tape_drivers/tape_drivers.h index 6eaa3777..0fe5f66f 100644 --- a/src/tape_drivers/tape_drivers.h +++ b/src/tape_drivers/tape_drivers.h @@ -150,11 +150,12 @@ static inline int _sense2errorcode(uint32_t sense, struct error_table *table, ch struct supported_device { char vendor_id[VENDOR_ID_LENGTH + 1]; char product_id[PRODUCT_ID_LENGTH + 1]; + int vendor_type; int drive_type; char product_name[PRODUCT_NAME_LENGTH + 1]; }; -#define TAPEDRIVE(v, p, t, n) &(struct supported_device){ v, p, t, n } +#define TAPEDRIVE(v, p, vt, dt, n) &(struct supported_device){ v, p, vt, dt, n } #define TAPE_FAMILY_MASK (0xF000) #define TAPE_FAMILY_ENTERPRISE (0x1000) @@ -178,7 +179,8 @@ enum { VENDOR_UNKNOWN = 0, VENDOR_IBM, VENDOR_HP, - VENDOR_QUANTUM, + VENDOR_QUANTUM, /* Quantum model C */ + VENDOR_QUANTUM_B, /* Quantum model B */ }; enum { diff --git a/src/tape_drivers/vendor_compat.c b/src/tape_drivers/vendor_compat.c index 0df8941f..f8a1ee33 100644 --- a/src/tape_drivers/vendor_compat.c +++ b/src/tape_drivers/vendor_compat.c @@ -281,35 +281,20 @@ struct error_table standard_tape_errors[] = { {0xFFFFFF, -EDEV_UNKNOWN, "Unknown Error code"}, }; -int get_vendor_id(char* vendor) -{ - if (!strncmp(vendor, IBM_VENDOR_ID, strlen(IBM_VENDOR_ID))) - return VENDOR_IBM; - else if (!strncmp(vendor, HP_VENDOR_ID, strlen(HP_VENDOR_ID))) - return VENDOR_HP; - else if (!strncmp(vendor, HPE_VENDOR_ID, strlen(HPE_VENDOR_ID))) - return VENDOR_HP; - else if (!strncmp(vendor, QUANTUM_VENDOR_ID, strlen(QUANTUM_VENDOR_ID))) - return VENDOR_QUANTUM; - else - return VENDOR_UNKNOWN; -} - -struct supported_device **get_supported_devs(int vendor) +struct supported_device **get_supported_devs(const char *vendor_str) { struct supported_device **cur = NULL; - switch (vendor) { - case VENDOR_IBM: - cur = ibm_supported_drives; - break; - case VENDOR_HP: - cur = hp_supported_drives; - break; - case VENDOR_QUANTUM: - cur = quantum_supported_drives; - break; - } + if (! strncmp(vendor_str, IBM_VENDOR_ID, strlen(IBM_VENDOR_ID))) + cur = ibm_supported_drives; + else if (! strncmp(vendor_str, HP_VENDOR_ID, strlen(HP_VENDOR_ID))) + cur = hp_supported_drives; + else if (! strncmp(vendor_str, HPE_VENDOR_ID, strlen(HPE_VENDOR_ID))) + cur = hp_supported_drives; + else if (! strncmp(vendor_str, TANDBERG_VENDOR_ID, strlen(TANDBERG_VENDOR_ID))) + cur = hp_supported_drives; + else if (! strncmp(vendor_str, QUANTUM_VENDOR_ID, strlen(QUANTUM_VENDOR_ID))) + cur = quantum_supported_drives; return cur; } diff --git a/src/tape_drivers/vendor_compat.h b/src/tape_drivers/vendor_compat.h index ebf8361c..fea2a29c 100644 --- a/src/tape_drivers/vendor_compat.h +++ b/src/tape_drivers/vendor_compat.h @@ -65,8 +65,7 @@ extern "C" { extern struct error_table standard_tape_errors[]; -int get_vendor_id(char* vendor); -struct supported_device **get_supported_devs(int vendor); +struct supported_device **get_supported_devs(const char *vendor_str); bool drive_has_supported_fw(int vendor, int drive_type, const unsigned char * const revision); unsigned char assume_cart_type(const unsigned char dc); int is_supported_tape(unsigned char type, unsigned char density, bool *is_worm); diff --git a/src/utils/Makefile.am b/src/utils/Makefile.am index 654a5cd0..918d8365 100644 --- a/src/utils/Makefile.am +++ b/src/utils/Makefile.am @@ -33,7 +33,7 @@ # OO_Copyright_END # -bin_PROGRAMS = mkltfs ltfsck +bin_PROGRAMS = mkltfs ltfsck ltfsindextool nobase_bin_SCRIPTS = ltfs_ordered_copy @@ -50,3 +50,9 @@ ltfsck_DEPENDENCIES = ../libltfs/libltfs.la ../../messages/libbin_ltfsck_dat.a ltfsck_LDADD = ../libltfs/libltfs.la ltfsck_LDFLAGS = @AM_LDFLAGS@ ../../messages/libbin_ltfsck_dat.a ltfsck_CPPFLAGS = @AM_CPPFLAGS@ -I .. -fPIC + +ltfsindextool_SOURCES = ltfsindextool.c +ltfsindextool_DEPENDENCIES = ../libltfs/libltfs.la ../../messages/libbin_ltfsindextool_dat.a ../../messages/libbin_mkltfs_dat.a +ltfsindextool_LDADD = ../libltfs/libltfs.la +ltfsindextool_LDFLAGS = @AM_LDFLAGS@ -L../../messages -lbin_ltfsindextool_dat +ltfsindextool_CPPFLAGS = @AM_CPPFLAGS@ -I .. -fPIC diff --git a/src/utils/ltfs_ordered_copy b/src/utils/ltfs_ordered_copy index 900ec7d1..8a5f4ea4 100755 --- a/src/utils/ltfs_ordered_copy +++ b/src/utils/ltfs_ordered_copy @@ -34,6 +34,7 @@ # OO_Copyright_END import sys +import errno import platform import os.path import argparse @@ -44,6 +45,11 @@ import threading from logging import getLogger, basicConfig, NOTSET, CRITICAL, ERROR, WARNING, INFO, DEBUG from collections import deque +def is_errno(num, names): + if not num in errno.errorcode: + return False + return errno.errorcode[num] in names + class CopyItem: """""" def __init__(self, src, dst, vea_pre, cp_attr, cp_xattr, logger): #initialization @@ -60,14 +66,14 @@ class CopyItem: def eval(self): #access to the extended attributes present in some operating systems/filesystems by xattr try: - self.vuuid = xattr.get(self.src, self.vea_pre + 'ltfs.volumeUUID') + self.vuuid = xattr.getxattr(self.src, self.vea_pre + 'ltfs.volumeUUID') except Exception as e: self.vuuid = '' return (self.vuuid, self.part, self.start) try: - self.part = xattr.get(self.src, self.vea_pre + 'ltfs.partition') - start_str = xattr.get(self.src, self.vea_pre + 'ltfs.startblock') + self.part = xattr.getxattr(self.src, self.vea_pre + 'ltfs.partition') + start_str = xattr.getxattr(self.src, self.vea_pre + 'ltfs.startblock') self.start = int(start_str) self.size = os.path.getsize(self.src) except Exception as e: @@ -89,8 +95,8 @@ class CopyItem: if self.cp_xattr: # Capture EAs of the source file src_attributes = {} - for key in xattr.list(self.src): - src_attributes[key] = xattr.get(self.src, key) + for key in xattr.listxattr(self.src): + src_attributes[key] = xattr.getxattr(self.src, key) # Set EAs of the destination file (_, filename) = os.path.split(self.src) for key in src_attributes: @@ -178,7 +184,7 @@ class CopyQueue: self.logger.log(NOTSET + 1, 'Making a directory {}'.format(d)) os.mkdir(d) except OSError as e: - if e.errno != 17: # Not EEXIST + if e.errno != errno.EEXIST: self.logger.error(str(e) + "\n") exit(1) @@ -251,7 +257,7 @@ LTFS_SIG_VEA='ltfs.softwareProduct' plat = platform.system() if plat == 'Linux': VEA_PREFIX='user.' -elif plat == 'Darwin': +elif plat == 'Darwin' or plat == 'FreeBSD': VEA_PREFIX='' else: sys.stderr.write("unsupported platform '{0}'\n".format(plat)) @@ -334,6 +340,9 @@ if args.DEST == None: logger.error('No destination is specified') exit(2) +if args.keep_tree is None: + args.keep_tree = '' + # Special case: # Copy source is only one file if args.recursive == False and len(args.SOURCE) == 1: @@ -359,9 +368,9 @@ if args.recursive == False and len(args.SOURCE) == 1: logger.log(NOTSET + 1, 'Checking destination is LTFS or not') direct_write_threads = 8 try: - sig = xattr.get(args.DEST, VEA_PREFIX + LTFS_SIG_VEA) + sig = xattr.getxattr(args.DEST, VEA_PREFIX + LTFS_SIG_VEA) - if sig.startswith("LTFS"): + if sig.startswith(b"LTFS"): logger.info("Destination {0} is LTFS".format(args.DEST)) direct_write_threads = 1 else: @@ -385,9 +394,6 @@ if len(args.SOURCE) == 0: args.SOURCE.append(line.rstrip('\r\n')) logger.log(NOTSET + 1, 'Source: {}'.format(args.SOURCE)) -if args.keep_tree is None: - args.keep_tree = '' - # Create the list of copy item copyq = CopyQueue(logger) for s in args.SOURCE: diff --git a/src/utils/ltfsck.c b/src/utils/ltfsck.c index fb1e4250..1ecc1787 100644 --- a/src/utils/ltfsck.c +++ b/src/utils/ltfsck.c @@ -99,27 +99,27 @@ enum { }; struct other_check_opts { - struct config_file *config; /**< Configurate data read from the global LTFS config file. */ - char *devname; /**< Device to format */ - char *backend_path; /**< Path to tape backend shared library */ - char *kmi_backend_name; /**< Name or path to the key manager interface backend library */ - int op_mode; /**< Operation mode */ - int search_mode; /**< Search mode for index */ - char *str_gen; /**< Rollback point specified by command line (generation)*/ - unsigned int point_gen; /**< Rollback point (generation) */ - bool erase_history; /**< overwrite existing data at rollback */ - bool recover_blocks; /**< Recover unreferenced blocks at the ends of the partitions? */ - bool deep_recovery; /**< Recover EOD missing cartridge? */ - int verbosity; /**< Print extra messages? */ - char *prg_name; /**< Program name */ - bool quiet; /**< Suppress information messages */ - bool trace; /**< Generate debug output */ - bool syslogtrace; /**< Generate debug output to stderr and syslog*/ - bool fulltrace; /**< Trace function calls */ - int traverse_mode; /**< Traverse strategy for listing index */ - bool full_index_info; /**< Print full index infomation in list mode */ - bool capture_index; /**< Capture index in list mode */ - bool salvage_points; /**< List rollback points from no-EOD cartridge? */ + struct config_file *config; /**< Configurate data read from the global LTFS config file. */ + char *devname; /**< Device to format */ + char *backend_path; /**< Path to tape backend shared library */ + char *kmi_backend_name; /**< Name or path to the key manager interface backend library */ + int op_mode; /**< Operation mode */ + int search_mode; /**< Search mode for index */ + char *str_gen; /**< Rollback point specified by command line (generation)*/ + unsigned int point_gen; /**< Rollback point (generation) */ + bool erase_history; /**< overwrite existing data at rollback */ + bool recover_blocks; /**< Recover unreferenced blocks at the ends of the partitions? */ + bool deep_recovery; /**< Recover EOD missing cartridge? */ + int verbosity; /**< Print extra messages? */ + char *prg_name; /**< Program name */ + bool quiet; /**< Suppress information messages */ + bool trace; /**< Generate debug output */ + bool syslogtrace; /**< Generate debug output to stderr and syslog*/ + bool fulltrace; /**< Trace function calls */ + int traverse_mode; /**< Traverse strategy for listing index */ + bool full_index_info; /**< Print full index infomation in list mode */ + char *capture_dir; /**< Capture index in list mode and it's directory */ + bool salvage_points; /**< List rollback points from no-EOD cartridge? */ }; struct index_info @@ -170,7 +170,7 @@ static struct option long_options[] = { {"generation", 1, 0, 'g'}, {"traverse", 1, 0, 'v'}, {"kmi-backend", 1, 0, '-'}, - {"capture-index", 0, 0, '+'}, + {"capture-index", 1, 0, '+'}, {"rollback" , 0, 0, 'r'}, {"no-rollback" , 0, 0, 'n'}, {"full-recovery" , 0, 0, 'f'}, @@ -356,7 +356,7 @@ int main(int argc, char **argv) break; case '+': opt.op_mode = MODE_LIST_POINT; - opt.capture_index = true; + opt.capture_dir = strdup(optarg); break; case 'r': opt.op_mode = MODE_ROLLBACK; @@ -717,6 +717,7 @@ int check_ltfs_volume(struct ltfs_volume *vol, struct other_check_opts *opt) return LTFSCK_UNCORRECTED; } else { print_criteria_info(vol); + ltfs_set_commit_message_reason(SYNC_CHECK, vol); ltfs_unmount(SYNC_CHECK, vol); ltfsmsg(LTFS_INFO, 16022I); return LTFSCK_CORRECTED; @@ -846,7 +847,8 @@ void _print_index_header(bool full_info) void _print_index(struct ltfs_volume *vol, struct index_info *list, struct other_check_opts *opt) { struct tm *t_st; - int i; + char *new_path = NULL; + int i, ret; if(!opt) return; @@ -941,8 +943,26 @@ void _print_index(struct ltfs_volume *vol, struct index_info *list, struct other else printf(" No commit message\n"); - if (opt->capture_index) - ltfs_save_index_to_disk(".", NULL, true, vol); + /* Rename reading xml to official name */ + if (vol->index_cache_path_r && opt->capture_dir) { + if (HAVE_BARCODE(vol)) + ret = asprintf(&new_path, "%s/%s-%d-%c.schema", opt->capture_dir, vol->label->barcode, + vol->index->generation, list->selfptr.partition); + else + ret = asprintf(&new_path, "%s/%s-%d-%c.schema", opt->capture_dir, vol->label->vol_uuid, + vol->index->generation, list->selfptr.partition); + + if (ret < 0) { + ltfsmsg(LTFS_ERR, 10001E, "_print_index: path"); + return; + } + + ret = rename(vol->index_cache_path_r, new_path); + if (ret < 0) { + ltfsmsg(LTFS_WARN, 16112W, vol->index_cache_path_r, new_path, errno); + } + free(new_path); + } return; } @@ -974,7 +994,7 @@ void print_index_array(struct ltfs_volume *vol, struct index_info *list, void *o while (cur) { _print_index(vol, cur, opt); - cur = cur-> next; + cur = cur->next; } return; @@ -1140,7 +1160,7 @@ int _rollback_ip(struct ltfs_volume *vol, struct other_check_opts *opt, struct t if (ret != LTFSCK_NO_ERRORS) ltfsmsg(LTFS_ERR, 16059E, ret); } else { - ret = ltfs_write_index(ltfs_ip_id(vol), SYNC_ROLLBACK, vol); + ret = ltfs_write_index(ltfs_ip_id(vol), SYNC_ROLLBACK, LTFS_FULL_INDEX, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 16060E, ret); ret = LTFSCK_OPERATIONAL_ERROR; @@ -1161,7 +1181,8 @@ int _rollback_dp(struct ltfs_volume *vol, struct other_check_opts *opt, struct t if (ret != LTFSCK_NO_ERRORS) ltfsmsg(LTFS_ERR, 16055E, ret); } else { - ret = ltfs_write_index(ltfs_dp_id(vol), SYNC_ROLLBACK, vol); + ltfs_set_commit_message_reason(SYNC_ROLLBACK, vol); + ret = ltfs_write_index(ltfs_dp_id(vol), SYNC_ROLLBACK, LTFS_FULL_INDEX, vol); if (ret < 0) { ltfsmsg(LTFS_ERR, 16056E, ret); ret = LTFSCK_OPERATIONAL_ERROR; @@ -1275,6 +1296,7 @@ int rollback(struct ltfs_volume *vol, struct other_check_opts *opt) r.current_pos = ltfs_get_index_selfpointer(vol); ltfsmsg(LTFS_DEBUG, 16081D, ltfs_get_index_generation(vol), (int)r.current_pos.partition, (unsigned long long)r.current_pos.block); + ltfs_set_commit_message_reason(SYNC_ROLLBACK, vol); ltfs_unmount(SYNC_ROLLBACK, vol); vol->index = NULL; } @@ -1286,11 +1308,11 @@ int rollback(struct ltfs_volume *vol, struct other_check_opts *opt) } /* Find target index */ - ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), opt->point_gen, + ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), opt->point_gen, false, search_index_by_gen, (void *)(&(r.target_info)), (void *)opt); if (ret == -LTFS_NO_INDEX) { if (opt->erase_history) { - ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), opt->point_gen, + ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), opt->point_gen, false, search_index_by_gen, (void *)(&(r.target_info)), (void *)opt); if (ret == -LTFS_NO_INDEX) { ltfsmsg(LTFS_ERR, 16072E, ret); @@ -1300,7 +1322,7 @@ int rollback(struct ltfs_volume *vol, struct other_check_opts *opt) return LTFSCK_OPERATIONAL_ERROR;; } } else { - ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), opt->point_gen, + ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), opt->point_gen, false, search_index_by_gen, (void *)(&(r.target_info)), (void *)opt); if (ret != LTFSCK_NO_ERRORS) { ltfsmsg(LTFS_ERR, 16072E, ret); @@ -1358,7 +1380,7 @@ int rollback(struct ltfs_volume *vol, struct other_check_opts *opt) int list_rollback_points_normal(struct ltfs_volume *vol, struct other_check_opts *opt) { - int ret = LTFSCK_NO_ERRORS; + int ret = LTFSCK_NO_ERRORS, fd = -1; /* Load tape and read labels */ ret = load_tape(vol); @@ -1378,14 +1400,41 @@ int list_rollback_points_normal(struct ltfs_volume *vol, struct other_check_opts } } + /* Configure index capturing */ + if (opt->capture_dir) { + /* Confirm the directory and construct read cache path */ + ret = asprintf(&vol->index_cache_path_r, "%s/reading_index.xml", opt->capture_dir); + + if (ret > 0) { + fd = open(vol->index_cache_path_r, O_WRONLY | O_BINARY | O_CREAT, + S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP); + if (fd < 0) { + ltfsmsg(LTFS_WARN, 16113W, opt->capture_dir, errno); + free(vol->index_cache_path_r); + vol->index_cache_path_r = NULL; + } else { + ltfsmsg(LTFS_INFO, 16114I, opt->capture_dir); + close(fd); + fd = -1; + } + } else { + ltfsmsg(LTFS_ERR, 10001E, "capture_dir"); + ltfs_volume_free(&vol); + return 1; + } + } else { + ltfsmsg(LTFS_INFO, 16115I); + } + + /* Print header */ _print_index_header(opt->full_index_info); /* read index from the index partition */ if(opt->traverse_mode == TRAVERSE_FORWARD) - ret = ltfs_traverse_index_forward(vol, ltfs_ip_id(vol), opt->point_gen, + ret = ltfs_traverse_index_forward(vol, ltfs_ip_id(vol), opt->point_gen, true, print_a_index_noheader, NULL, (void *)opt); else - ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), opt->point_gen, + ret = ltfs_traverse_index_backward(vol, ltfs_ip_id(vol), opt->point_gen, true, print_a_index_noheader, NULL, (void *)opt); if (ret != LTFSCK_NO_ERRORS) { ltfsmsg(LTFS_ERR, 16075E, ret); @@ -1394,10 +1443,10 @@ int list_rollback_points_normal(struct ltfs_volume *vol, struct other_check_opts /* read index from the data partition */ if(opt->traverse_mode == TRAVERSE_FORWARD) - ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), opt->point_gen, + ret = ltfs_traverse_index_forward(vol, ltfs_dp_id(vol), opt->point_gen, true, print_a_index_noheader, NULL, (void *)opt); else - ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), opt->point_gen, + ret = ltfs_traverse_index_backward(vol, ltfs_dp_id(vol), opt->point_gen, true, print_a_index_noheader, NULL, (void *)opt); if (ret != LTFSCK_NO_ERRORS) { ltfsmsg(LTFS_ERR, 16076E, ret); @@ -1433,7 +1482,7 @@ int list_rollback_points_no_eod(struct ltfs_volume *vol, struct other_check_opts /* Read index from the data partition */ /* We don't need to read IP because index in DP is always newer (or same) than IP in case of WORM */ - ret = ltfs_traverse_index_no_eod(vol, ltfs_dp_id(vol), opt->point_gen, + ret = ltfs_traverse_index_no_eod(vol, ltfs_dp_id(vol), opt->point_gen, true, print_a_index_noheader, NULL, (void *)opt); if (ret != LTFSCK_NO_ERRORS) { ltfsmsg(LTFS_ERR, 16076E, ret); @@ -1493,7 +1542,7 @@ int _ltfsck_validate_options(struct other_check_opts *opt) else if (opt->traverse_mode == TRAVERSE_BACKWARD) ltfsmsg(LTFS_INFO, 16084I); - if (opt->capture_index && opt->search_mode == SEARCH_BY_GEN) { + if (opt->capture_dir && opt->search_mode == SEARCH_BY_GEN) { if (!opt->str_gen) { ltfsmsg(LTFS_ERR, 16004E); return LTFSCK_USAGE_SYNTAX_ERROR; diff --git a/src/utils/ltfsindextool.c b/src/utils/ltfsindextool.c new file mode 100644 index 00000000..c4d47a64 --- /dev/null +++ b/src/utils/ltfsindextool.c @@ -0,0 +1,775 @@ +/* +** +** OO_Copyright_BEGIN +** +** +** Copyright 2010, 2020 IBM Corp. All rights reserved. +** +** Redistribution and use in source and binary forms, with or without +** modification, are permitted provided that the following conditions +** are met: +** 1. Redistributions of source code must retain the above copyright +** notice, this list of conditions and the following disclaimer. +** 2. Redistributions in binary form must reproduce the above copyright +** notice, this list of conditions and the following disclaimer in the +** documentation and/or other materials provided with the distribution. +** 3. Neither the name of the copyright holder nor the names of its +** contributors may be used to endorse or promote products derived from +** this software without specific prior written permission. +** +** THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' +** AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +** IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +** ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +** LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +** CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +** SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +** INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +** CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +** ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +** POSSIBILITY OF SUCH DAMAGE. +** +** +** OO_Copyright_END +** +************************************************************************************* +** +** COMPONENT NAME: IBM Linear Tape File System +** +** FILE NAME: utils/ltfsindextool.c +** +** DESCRIPTION: The low level index tool +** +** AUTHORS: Atsushi Abe +** piste.2750@gmail.com +** +************************************************************************************* +*/ + +#ifdef mingw_PLATFORM +#include "libltfs/arch/win/win_util.h" +#else +#include +#endif /* mingw_PLATFORM */ + +#include +#include "libltfs/ltfs_fuse_version.h" +#include + +#include "libltfs/ltfs.h" +#include "libltfs/xml_libltfs.h" +#include "ltfs_copyright.h" +#include "libltfs/plugin.h" +#include "libltfs/kmi.h" +#include "libltfs/tape.h" + +volatile char *copyright = LTFS_COPYRIGHT_0"\n"LTFS_COPYRIGHT_1"\n"LTFS_COPYRIGHT_2"\n" \ + LTFS_COPYRIGHT_3"\n"LTFS_COPYRIGHT_4"\n"LTFS_COPYRIGHT_5"\n"; + +#ifdef __APPLE__ +#include "libltfs/arch/osx/osx_string.h" +#endif + +#ifdef mingw_PLATFORM +char *bin_ltfsindextool_dat; +#else +extern char bin_ltfsindextool_dat[]; +#endif + +typedef enum { + OP_CHECK, + OP_CAPTURE, +} operation_mode_t; + +struct indextool_opts { + operation_mode_t mode; /**< Operation mode */ + char *filename; /**< Index filename to check in check mode */ + char *devname; /**< Device name to capture index in capture mode */ + int partition; /**< partition to operate */ + uint64_t start_pos; /**< start position */ + char *out_dir; /**< Output dir for captured indexes */ + unsigned long blocksize; /**< Nominal tape block size */ + struct config_file *config; /**< Configuration data read from the global LTFS config file */ + char *backend_path; /**< Path to tape backend shared library */ + char *kmi_backend_name; /**< Name or path to the key manager interface backend library */ + bool quiet; /**< Quiet mode indicator */ + bool trace; /**< Debug mode indicator */ + bool syslogtrace; /**< Generate debug output to stderr and syslog*/ +}; + +/* Command line options */ +#define PART_BOTH (-1) +#define START_POS (5) +#define OUTPUT_DIR "." +#define KEY_MAX_OFFSET (0x30) + +static const char *short_options = "i:e:d:p:s:o:b:qthV"; +static struct option long_options[] = { + {"config", 1, 0, 'i'}, + {"backend", 1, 0, 'e'}, + {"device", 1, 0, 'd'}, + {"partition", 1, 0, 'p'}, + {"start-pos", 1, 0, 's'}, + {"output-dir", 1, 0, '^'}, + {"blocksize", 1, 0, 'b'}, + {"kmi-backend", 1, 0, '-'}, + {"quiet", 0, 0, 'q'}, + {"trace", 0, 0, 't'}, + {"syslogtrace", 0, 0, '!'}, + {"help", 0, 0, 'h'}, + {"version", 0, 0, 'V'}, + {0, 0, 0, 0} +}; + +/* Private functions */ +static inline int _open_output_file(tape_partition_t part, + tape_block_t start_pos, + char *base_path) +{ + int ret; + char *fname = NULL; + + ret = asprintf(&fname, "%s/ltfs-index-%u-%llu.xml", base_path, + (unsigned int) part, (unsigned long long)start_pos); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19532E); + return -1; + } + + ltfsmsg(LTFS_INFO, 19547I, fname); + ret = open(fname, O_WRONLY | O_CREAT | O_TRUNC, 0666); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19533E, fname, errno); + } + + free(fname); + + return ret; +} + +static inline void _close_output_file(int fd) +{ + fsync(fd); + close(fd); +} + +/** Capture indexes on the partition. + * + */ +static int ltfs_capture_index_raw(tape_partition_t part, + tape_block_t start_pos, + int blocksize, + char *base_path, + struct ltfs_volume *vol) +{ + int ret = 0, fd = -1; + ssize_t nread, nwrite, index_len = 0; + struct tc_position pos; + char *buf = NULL, *key = NULL, check_buf[KEY_MAX_OFFSET + 1]; + + pos.partition = part; + pos.block = start_pos; + + buf = malloc(blocksize); + if (!buf) { + ltfsmsg(LTFS_ERR, 19516E); + return -LTFS_NO_MEMORY; + } + + ret = tape_seek(vol->device, &pos); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19517E, (unsigned int)part, (unsigned long long)start_pos, ret); + return ret; + } + + while (ret == 0) { + index_len = 0; + + ret = tape_get_position(vol->device, &pos); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19518E, ret); + break; + } + + nread = tape_read(vol->device, buf, blocksize, true, vol->kmi_handle); + if (nread < 0) { + ltfsmsg(LTFS_ERR, 19519E, + (unsigned int)pos.partition, (unsigned long long)pos.block, nread); + ret = nread; + break; + } + pos.block++; + + memset(check_buf, 0x00, KEY_MAX_OFFSET + 1); + strncpy(check_buf, buf, KEY_MAX_OFFSET); + key = strstr(check_buf, "device, buf, blocksize, true, vol->kmi_handle); + if (nread == -EDEV_EOD_DETECTED) { + ret = nread; + ltfsmsg(LTFS_ERR, 19538E, + (unsigned int)pos.partition, + (unsigned long long)(pos.block)); + ltfsmsg(LTFS_INFO, 19539I, (unsigned long long)index_len); + break; + } + + if (nread > 0) { + /* Write a block to the file */ + nwrite = write(fd, buf, nread); + if (nwrite == nread) { + index_len += nread; + } else { + ltfsmsg(LTFS_ERR, 19536E, nwrite, errno); + _close_output_file(fd); + return -LTFS_CACHE_IO; + } + pos.block++; + } else if (!nread) { + /* Detect a FM (the end of the index), do nothing */ + ltfsmsg(LTFS_INFO, 19537I, + (unsigned int)pos.partition, + (unsigned long long)(pos.block)); + ltfsmsg(LTFS_INFO, 19539I, (unsigned long long)index_len); + break; + } else { + ltfsmsg(LTFS_ERR, 19519E, + (unsigned int)pos.partition, (unsigned long long)pos.block, nread); + ret = nread; + break; + } + } + + _close_output_file(fd); + } else { + /* seek to next FM */ + if (key) + ltfsmsg(LTFS_INFO, 19530I, + (unsigned int)pos.partition, + (unsigned long long)(pos.block - 1), + (int)(key - buf)); + else + ltfsmsg(LTFS_INFO, 19530I, + (unsigned int)pos.partition, + (unsigned long long)(pos.block - 1), + (int)0); + + /* Do nothig at (nread == 0) because tape hits a FM */ + if (nread > 0) { + ret = tape_spacefm(vol->device, 1); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19531E, (unsigned int)part, (unsigned long long)start_pos, ret); + break; + } + } + } + } + + if (ret == -EDEV_EOD_DETECTED) { + ret = tape_get_position(vol->device, &pos); + if (!ret) { + ltfsmsg(LTFS_INFO, 19534I,(unsigned int)pos.partition, (unsigned long long)pos.block); + } else { + ltfsmsg(LTFS_INFO, 19535I, ret); + } + ret = 0; + } + + if (buf) + free(buf); + + return ret; +} + +static int _capture(struct indextool_opts *opt, struct ltfs_volume *vol) +{ + int r, ret = 0; + tape_partition_t p; + + if (opt->partition == PART_BOTH) { + ltfsmsg(LTFS_INFO, 19504I); + for (p = 0; p < 2; p++) { + r = ltfs_capture_index_raw(p, 5, opt->blocksize, opt->out_dir, vol); + if (!ret) + ret = r; + } + } else { + ltfsmsg(LTFS_INFO, 19505I, (unsigned int)opt->partition, (unsigned long long)opt->start_pos); + ret = ltfs_capture_index_raw(opt->partition, opt->start_pos, opt->blocksize, opt->out_dir, vol); + } + + return ret; +} + +static int _indextool_validate_options(char *prg_name, struct indextool_opts *opt) +{ + ltfsmsg(LTFS_DEBUG, 19525D); + + /* Validate filename and devname and decide a operation mode*/ + if (opt->filename) { + opt->mode = OP_CHECK; + } else if (opt->devname) { + opt->mode = OP_CAPTURE; + } else { + ltfsmsg(LTFS_ERR, 19526E); + return 1; + } + + /* Validate partition */ + if (opt->partition != PART_BOTH && opt->partition != 0 && opt->partition != 1) { + ltfsmsg(LTFS_ERR, 19540E); + return 1; + } + + /* Validate start position */ + if (opt->start_pos < START_POS) { + ltfsmsg(LTFS_ERR, 19548E, (unsigned long long)opt->start_pos); + return 1; + } + + ltfsmsg(LTFS_DEBUG, 19527D); + return 0; +} + +static int check_index(struct ltfs_volume *vol, struct indextool_opts *opt, void *args) +{ + int ret = 0; + + ltfsmsg(LTFS_INFO, 19543I, opt->filename); + + vol->label->blocksize = opt->blocksize; + ret = xml_schema_from_file(opt->filename, vol->index, vol); + + if (!ret) { + ltfsmsg(LTFS_INFO, 19544I); + } else { + ltfsmsg(LTFS_ERR, 19545E, ret); + } + + return ret; +} + +static int capture_index(struct ltfs_volume *vol, struct indextool_opts *opt, void *args) +{ + int ret = INDEXTOOL_OPERATIONAL_ERROR; + struct libltfs_plugin backend; /* tape driver backend */ + struct libltfs_plugin kmi; /* key manager interface backend */ + + /* load the backend, open the tape device, and load a tape */ + ltfsmsg(LTFS_DEBUG, 19506D); + ret = plugin_load(&backend, "tape", opt->backend_path, opt->config); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19508E, opt->backend_path); + return INDEXTOOL_OPERATIONAL_ERROR; + } + if (opt->kmi_backend_name) { + ret = plugin_load(&kmi, "kmi", opt->kmi_backend_name, opt->config); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19509E, opt->kmi_backend_name); + return INDEXTOOL_OPERATIONAL_ERROR; + } + } + ret = ltfs_device_open(opt->devname, backend.ops, vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19510E, opt->devname, ret); + ret = INDEXTOOL_OPERATIONAL_ERROR; + goto out_unload_backend; + } + ret = ltfs_parse_tape_backend_opts(args, vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19513E); + goto out_unload_backend; + } + if (opt->kmi_backend_name) { + ret = kmi_init(&kmi, vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19511E, opt->devname, ret); + goto out_unload_backend; + } + + ret = ltfs_parse_kmi_backend_opts(args, vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19512E); + goto out_unload_backend; + } + + ret = tape_clear_key(vol->device, vol->kmi_handle); + if (ret < 0) + goto out_unload_backend; + } + + { + int i = 0; + struct fuse_args *a = args; + + for (i = 0; i < a->argc && a->argv[i]; ++i) { + if (!strcmp(a->argv[i], "-o")) { + ltfsmsg(LTFS_ERR, 19514E, a->argv[i], a->argv[i + 1] ? a->argv[i + 1] : ""); + ret = INDEXTOOL_USAGE_SYNTAX_ERROR; + goto out_unload_backend; + } + } + } + + ltfs_load_tape(vol); + ret = ltfs_wait_device_ready(vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19515E); + ret = INDEXTOOL_OPERATIONAL_ERROR; + goto out_close; + } + + vol->append_only_mode = false; + vol->set_pew = false; + ret = ltfs_setup_device(vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19515E); + ret = INDEXTOOL_OPERATIONAL_ERROR; + goto out_close; + } + ltfsmsg(LTFS_DEBUG, 19507D); + + /* Capture_index */ + ret = _capture(opt, vol); + + /* close the tape device and unload the backend */ + ltfsmsg(LTFS_DEBUG, 19520D); + +out_close: + ltfs_device_close(vol); + ltfs_volume_free(&vol); + ltfs_unset_signal_handlers(); + if (ret == INDEXTOOL_NO_ERRORS) + ltfsmsg(LTFS_DEBUG, 19522D); +out_unload_backend: + if (ret == INDEXTOOL_NO_ERRORS) { + ret = plugin_unload(&backend); + if (ret < 0) + ltfsmsg(LTFS_WARN, 19521W); + if (opt->kmi_backend_name) { + ret = plugin_unload(&kmi); + if (ret < 0) + ltfsmsg(LTFS_WARN, 19528W); + } + ret = INDEXTOOL_NO_ERRORS; + } else { + plugin_unload(&backend); + if (opt->kmi_backend_name) + plugin_unload(&kmi); + } + + if (ret == INDEXTOOL_NO_ERRORS) + ltfsmsg(LTFS_INFO, 19524I); + else + ltfsmsg(LTFS_INFO, 19523I, ret); + + return ret; +} + +static void show_usage(char *appname, struct config_file *config, bool full) +{ + struct libltfs_plugin backend; + const char *default_backend; + char *devname = NULL; + + default_backend = config_file_get_default_plugin("tape", config); + if (default_backend && plugin_load(&backend, "tape", default_backend, config) == 0) { + devname = strdup(ltfs_default_device_name(backend.ops)); + plugin_unload(&backend); + } + + if (! devname) + devname = strdup(""); + + fprintf(stderr, "\n"); + ltfsresult(19900I, appname); /* Usage: %s */ + fprintf(stderr, "\n"); + ltfsresult(19901I); /* Available options are: */ + ltfsresult(19902I); /* -d, --device= */ + ltfsresult(19903I); /* -p, --partition=<0|1> */ + ltfsresult(19904I, START_POS); /* -s, --start-pos */ + ltfsresult(19905I, OUTPUT_DIR); /* -output-dir */ + ltfsresult(19906I, LTFS_DEFAULT_BLOCKSIZE); /* -b, --blocksize */ + ltfsresult(19907I, LTFS_CONFIG_FILE); /* -i, --config= */ + ltfsresult(19908I, default_backend); /* -e, --backend */ + ltfsresult(19909I, config_file_get_default_plugin("kmi", config)); /* --kmi-backend */ + ltfsresult(19910I); /* -q, --quiet */ + ltfsresult(19911I); /* -t, --trace */ + ltfsresult(19912I); /* -V, --version */ + ltfsresult(19913I); /* -h, --help */ + fprintf(stderr, "\n"); + plugin_usage(appname, "driver", config); + fprintf(stderr, "\n"); + plugin_usage(appname, "kmi", config); + fprintf(stderr, "\n"); + ltfsresult(19914I); /* Usage example: */ + ltfsresult(19915I, appname, devname, 0); + free(devname); +} + +/* Main routine */ +int main(int argc, char **argv) +{ + struct ltfs_volume *vol; + struct indextool_opts opt; + int ret, log_level, syslog_level, i, cmd_args_len; + char *lang, *cmd_args; + const char *config_file = NULL; + void *message_handle; + + int fuse_argc = argc; + char **fuse_argv = calloc(fuse_argc, sizeof(char *)); + if (! fuse_argv) { + return INDEXTOOL_OPERATIONAL_ERROR; + } + for (i = 0; i < fuse_argc; ++i) { + fuse_argv[i] = strdup(argv[i]); + if (! fuse_argv[i]) { + return INDEXTOOL_OPERATIONAL_ERROR; + } + } + struct fuse_args args = FUSE_ARGS_INIT(fuse_argc, fuse_argv); + + /* Check for LANG variable and set it to en_US.UTF-8 if it is unset. */ + lang = getenv("LANG"); + if (! lang) { + fprintf(stderr, "LTFS9015W Setting the locale to 'en_US.UTF-8'. If this is wrong, please set the LANG environment variable before starting mkltfs.\n"); + ret = setenv("LANG", "en_US.UTF-8", 1); + if (ret) { + fprintf(stderr, "LTFS9016E Cannot set the LANG environment variable\n"); + return INDEXTOOL_OPERATIONAL_ERROR; + } + } + + /* Start up libltfs with the default logging level. */ +#ifndef mingw_PLATFORM + openlog("ltfsindextool", LOG_PID, LOG_USER); +#endif + ret = ltfs_init(LTFS_INFO, true, false); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 10000E, ret); + return INDEXTOOL_OPERATIONAL_ERROR; + } + + /* Setup signal handler to terminate cleanly */ + ret = ltfs_set_signal_handlers(); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 10013E); + return INDEXTOOL_OPERATIONAL_ERROR; + } + + /* Register messages with libltfs */ + ret = ltfsprintf_load_plugin("bin_ltfsindextool", bin_ltfsindextool_dat, &message_handle); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 10012E, ret); + return INDEXTOOL_OPERATIONAL_ERROR; + } + + /* Set up empty options and load the configuration file. */ + memset(&opt, 0, sizeof(struct indextool_opts)); + opt.blocksize = LTFS_DEFAULT_BLOCKSIZE; + opt.partition = PART_BOTH; + opt.start_pos = START_POS; + opt.out_dir = OUTPUT_DIR; + + /* Check for a config file path given on the command line */ + while (true) { + int option_index = 0; + int c = getopt_long(argc, argv, short_options, long_options, &option_index); + if (c == -1) + break; + if (c == 'i') { + config_file = strdup(optarg); + break; + } + } + + /* Load configuration file */ + ret = config_file_load(config_file, &opt.config); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 10008E, ret); + return INDEXTOOL_OPERATIONAL_ERROR; + } + + /* Parse all command line arguments */ + + optind = 1; + int num_of_o = 0; + while (true) { + int option_index = 0; + int c = getopt_long(argc, argv, short_options, long_options, &option_index); + if (c == -1) + break; + + switch (c) { + case 'i': + break; + case 'e': + free(opt.backend_path); + opt.backend_path = strdup(optarg); + break; + case 'd': + opt.devname = strdup(optarg); + break; + case 'p': + opt.partition = atoi(optarg); + break; + case 's': + opt.start_pos = strtoull(optarg, NULL, 0); + break; + case '^': + opt.out_dir = strdup(optarg); + break; + case 'b': + opt.blocksize = atoi(optarg); + break; + case '-': + opt.kmi_backend_name = strdup(optarg); + break; + case 'q': + opt.quiet = true; + break; + case 't': + opt.trace = true; + break; + case '!': + opt.syslogtrace = true; + break; + case 'h': + show_usage(argv[0], opt.config, false); + return 0; + case 'V': + ltfsresult(19546I, "ltfsindextool", PACKAGE_VERSION); + ltfsresult(19546I, "LTFS Format Specification", LTFS_INDEX_VERSION_STR); + return 0; + case '?': + default: + show_usage(argv[0], opt.config, false); + return INDEXTOOL_USAGE_SYNTAX_ERROR; + } + } + + if(argv[optind + num_of_o]) + opt.filename = strdup(argv[optind + num_of_o]); + + if(_indextool_validate_options(argv[0], &opt)) { + return PROG_USAGE_SYNTAX_ERROR; + } + + /* Pick up default backend if one wasn't specified before */ + if (! opt.backend_path) { + const char *default_backend = config_file_get_default_plugin("tape", opt.config); + if (! default_backend) { + ltfsmsg(LTFS_ERR, 10009E); + return INDEXTOOL_OPERATIONAL_ERROR; + } + opt.backend_path = strdup(default_backend); + } + if (! opt.kmi_backend_name) { + const char *default_backend = config_file_get_default_plugin("kmi", opt.config); + if (default_backend) + opt.kmi_backend_name = strdup(default_backend); + else + opt.kmi_backend_name = strdup("none"); + } + if (opt.kmi_backend_name && strcmp(opt.kmi_backend_name, "none") == 0) + opt.kmi_backend_name = NULL; + + /* Set the logging level */ + if (opt.quiet && opt.trace) { + ltfsmsg(LTFS_ERR, 9012E); + show_usage(argv[0], opt.config, false); + return 1; + } else if (opt.quiet) { + log_level = LTFS_WARN; + syslog_level = LTFS_NONE; + } else if (opt.trace) { + log_level = LTFS_DEBUG; + syslog_level = LTFS_NONE; + } else if (opt.syslogtrace) + log_level = syslog_level = LTFS_DEBUG; + else { + log_level = LTFS_INFO; + syslog_level = LTFS_NONE; + } + + ltfs_set_log_level(log_level); + ltfs_set_syslog_level(syslog_level); + + /* Starting ltfsindextool */ + ltfsmsg(LTFS_INFO, 19500I, PACKAGE_NAME, PACKAGE_VERSION, log_level); + + /* Show command line arguments */ + for (i = 0, cmd_args_len = 0 ; i < argc; i++) { + cmd_args_len += strlen(argv[i]) + 1; + } + cmd_args = calloc(1, cmd_args_len + 1); + if (!cmd_args) { + /* Memory allocation failed */ + ltfsmsg(LTFS_ERR, 10001E, "ltfsindextool (arguments)"); + return INDEXTOOL_OPERATIONAL_ERROR; + } + strcat(cmd_args, argv[0]); + for (i = 1; i < argc; i++) { + strcat(cmd_args, " "); + strcat(cmd_args, argv[i]); + } + ltfsmsg(LTFS_INFO, 19542I, cmd_args); + free(cmd_args); + + /* Show build time information */ + ltfsmsg(LTFS_INFO, 19502I, BUILD_SYS_FOR); + ltfsmsg(LTFS_INFO, 19503I, BUILD_SYS_GCC); + + /* Show run time information */ + show_runtime_system_info(); + + /* Actually mkltfs logic starts here */ + ret = ltfs_volume_alloc("dummy", &vol); + if (ret < 0) { + ltfsmsg(LTFS_ERR, 19501E); + return INDEXTOOL_OPERATIONAL_ERROR; + } + + switch (opt.mode) { + case OP_CHECK: + ret = check_index(vol, &opt, &args); + break; + case OP_CAPTURE: + ret = capture_index(vol, &opt, &args); + break; + default: + ltfsmsg(LTFS_ERR, 19541E); + ret = PROG_USAGE_SYNTAX_ERROR; + break; + } + + /* Cleaning up */ + free(opt.backend_path); + free(opt.kmi_backend_name); + free(opt.devname); + config_file_free(opt.config); + ltfsprintf_unload_plugin(message_handle); + ltfs_finish(); + + return ret; +}