Commit 7f04be37 authored by Paal Kvamme's avatar Paal Kvamme
Browse files

Add IZgyWriter::reopen() to allow (with some restrictions) an existing file to be modified.

parent 5a081fb7
......@@ -1298,6 +1298,118 @@ IZgyWriter::open(const ZgyWriterArgs& args)
return std::shared_ptr<IZgyWriter>(new Impl::ZgySafeWriter(unsafe));
}
/**
* \brief Open an existing ZGY file for writing.
*
* \details
* There are several restrictions on using this feature.
* Some of those might be relaxed in the future.
*
* \li size, bricksize, datatype, and datarange are not allowed to change.
*
* \li filename, iocontext, compressor, and lodcompressor are not stored
* in the file itself. So the caller needs to re-specify them.
*
* \li ilstart, ilinc, xlstart, xlinc, zstart, zinc, zunit, hunit, corners
* can all be changed but note that this can only be done in the
* actual call to reopen().
*
* \li The operation is expensive, even when the only change is to
* update some metadata. The current implementation will delete
* and re-create the file behind the scenes.
*
* \li The file to be opened cannot contain any bulk data. The only
* exception is that the entire file may have been initialized to some
* arbitrary constant.
*
* \li If the file to be updated is stored on a cloud back-end then it must
* have been created by OpenZGY and written directly to the cloud.
* The reason: Even though the ZGY-Public and OpenZGY file formats are
* identical, the data on the cloud can be split into multiple chunks
* which are logically treated as a single byte stream. The choice
* of where to split differs between OpenZGY, ZGY-Public, and uploads
* done by sdutil. This restriction is currently not active but it will
* become active when a better implementation (removing most of the
* other rules) has been coded.
*
* \li If the file does contain bulk data (which currently isn't allowed)
* then it is strongly advised to not change from compressed to
* uncompressed or vice versa. Even though this might not be checked.
* If you ignore this advice then he new compression mode will only
* apply to subsequent writes.
*
* \li If the file does contain bulk data (which currently isn't allowed)
* then it is strongly advised to not update any already existing data.
* Including overwrite due to a read/modify/write cycle. With a cloud
* backend this could waste storage and might also be refused at runtime.
*
* \li A file that has never been written to may end up as a file that
* has been explicitly written as the default (usually zero) value.
* The OpenZGY api hides that distinction anyway. So it shouldn't matter.
*
* \li The operation can leave the file unusable or even deleted if some
* error occurs. Since we are creating a brand new file it would be
* possible to create this with a new name and only rename it and
* discard the original once we know that the new file is good.
* But, (a) rename on the cloud is difficult and (b) once this code
* gets implemented properly that trick isn't possible anyway.
*
* \li The current implementation is incomplete in the sense that it
* requires the caller to re-specify the unit, annotation, and world
* coordinate information. If not specified they will revert to the
* default value instead of being left unchanged. That particular
* problem is fairly easy to fix. Stay tuned.
*/
std::shared_ptr<IZgyWriter>
IZgyWriter::reopen(const ZgyWriterArgs& args)
{
// Get all existing information about the file. Including the value
// of all samples, if and only if all the samples are the same.
std::shared_ptr<IZgyReader> reader = IZgyReader::open(args._filename, args._iocontext);
ZgyWriterArgs newargs = ZgyWriterArgs()
.metafrom(reader)
.filename(args._filename);
std::pair<bool,double> value =
reader->readconst(size3i_t{0,0,0}, reader->size(), 0, true);
reader->close();
reader.reset();
if (!value.first)
throw Errors::ZgyUserError("Cannot re-open a file that already contains data.");
// Now create a new file with the same name, making it look like we
// have just opened the old file for update. At the file back-end
// level the file is either deleted and then re-created or it is
// truncated. Either way all the information in the file is lost. At
// the ZgyWriter level the metadata is initialized to what we
// salvaged from the original. Which makes this appear to be an
// update operation. As for the bulk data, we only copy that if all
// the samples had the same value. If the input had never been
// written the new file will be explicitly written as the default
// (usually zero) value. This should not make any difference because
// OpenZGY hides that distinction anyway.
newargs
// .merge(update_args) // Not implemented yet!
// TODO: Should only be merged if explicitly set.
.filename(args._filename)
.iocontext(args._iocontext)
.compressor(args._compressor)
.lodcompressor(args._lodcompressor)
.ilstart(args._annotstart[0]).ilinc(args._annotinc[0])
.xlstart(args._annotstart[1]).xlinc(args._annotinc[1])
.zstart(args._zstart).zinc(args._zinc)
.corners(args._corners)
.hunit(args._hunitdim, args._hunitname, args._hunitfactor)
.zunit(args._zunitdim, args._zunitname, args._zunitfactor);
std::shared_ptr<IZgyWriter> writer = IZgyWriter::open(newargs);
// Cast is ok because all supported data types (int8, int16, float)
// can be cast to a single precision float without loss of accuracy.
const float floatval = static_cast<float>(value.second);
writer->writeconst(size3i_t{0,0,0}, writer->size(), &floatval);
// At this point it ought to look like an existing file was opened for update.
return writer;
}
std::shared_ptr<IZgyUtils>
IZgyUtils::utils(const std::string& prefix, const IOContext* iocontext)
{
......
......@@ -424,6 +424,7 @@ private:
friend class Impl::ZgyWriter; // Processing the high level parts.
friend class ::Test::ZgyWriterMock;
friend class Impl::EnumMapper; // Mapping the rest to the internal struct.
friend class IZgyWriter; // For IZgyWriter::reopen()
std::string _filename;
const IOContext *_iocontext; // TODO-Worry: Sure instance is short lived?
compressor_t _compressor;
......@@ -790,6 +791,8 @@ public:
/// \endcond
/// \brief Create a ZGY file and open it for writing.
static std::shared_ptr<IZgyWriter> open(const ZgyWriterArgs& args);
/// \brief Open an existing ZGY file for writing.
static std::shared_ptr<IZgyWriter> reopen(const ZgyWriterArgs& args);
};
/**
......
......@@ -594,18 +594,20 @@ void test_ZgyWriterArgs()
TEST_CHECK(ss.str().find("\"ms\"") != std::string::npos);
}
void test_write()
static void do_write_once(const std::string& filename)
{
LocalFileAutoDelete lad("testfile.zgy");
ZgyWriterArgs args = ZgyWriterArgs()
.filename(lad.name())
.filename(filename)
.size(33, 28, 92)
.datatype(SampleDataType::int16)
.datarange(-32768,+32767)
.ilstart(1).ilinc(2)
.xlstart(500).xlinc(5)
.zstart(100).zinc(4)
.zunit(UnitDimension::time, "ms", 1000);
.hunit(UnitDimension::length, "m", 1)
.zunit(UnitDimension::time, "ms", 1000)
.corners(ZgyWriterArgs::corners_t{5,7,5,107,205,7,205,107});
std::shared_ptr<OpenZGY::IZgyWriter> writer = OpenZGY::IZgyWriter::open(args);
std::vector<float> data(2*3*4, -1000);
const OpenZGY::IZgyWriter::size3i_t origin{0,0,0};
......@@ -616,14 +618,19 @@ void test_write()
writer->write(origin, count, data.data());
writer->finalize(std::vector<OpenZGY::DecimationType>{}, nullptr);
writer->close();
}
std::shared_ptr<OpenZGY::IZgyReader> reader = OpenZGY::IZgyReader::open(lad.name());
static void do_check_written(const std::string& filename)
{
std::shared_ptr<OpenZGY::IZgyReader> reader = OpenZGY::IZgyReader::open(filename);
if (verbose()) {
std::cout << "\n";
dump_api(reader, std::cout);
//reader->dump(std::cout);
}
std::unique_ptr<float[]> checkdata(new float[64*64*64]);
const OpenZGY::IZgyWriter::size3i_t origin{0,0,0};
const OpenZGY::IZgyWriter::size3i_t bsize{64,64,64};
reader->read(origin, bsize, checkdata.get(), 0);
TEST_CHECK(checkdata[0] == -1000);
TEST_CHECK(checkdata[63] == 42);
......@@ -633,9 +640,90 @@ void test_write()
TEST_CHECK(h.samplecount == 33*28*92);
TEST_CHECK(h.bins[124] == 2*3*4);
TEST_CHECK(h.bins[128] == 33*28*92 - 2*3*4);
const ZgyWriterArgs::corners_t corners = reader->corners();
const double eps = 1.0e-10;
TEST_CHECK(fabs(corners[0][0] - 5) <= eps);
TEST_CHECK(fabs(corners[0][1] - 7) <= eps);
TEST_CHECK(fabs(corners[1][0] - 5) <= eps);
TEST_CHECK(fabs(corners[1][1] - 107) <= eps);
TEST_CHECK(fabs(corners[2][0] - 205) <= eps);
TEST_CHECK(fabs(corners[2][1] - 7) <= eps);
TEST_CHECK(fabs(corners[3][0] - 205) <= eps);
TEST_CHECK(fabs(corners[3][1] - 107) <= eps);
TEST_CHECK(reader->size() == (OpenZGY::IZgyWriter::size3i_t{33, 28, 92}));
TEST_CHECK(reader->datatype() == SampleDataType::int16);
TEST_CHECK(reader->hunitname() == "m");
TEST_CHECK(reader->zunitname() == "ms");
TEST_CHECK(reader->annotstart()[0] == 1);
TEST_CHECK(reader->annotstart()[1] == 500);
TEST_CHECK(reader->zstart() == 100);
reader->close();
}
/**
* As do_write_once() but the file gets opened, closed, and opened again.
*/
static void do_write_twice(const std::string& filename)
{
ZgyWriterArgs firstargs = ZgyWriterArgs()
.filename(filename)
.size(33, 28, 92)
.bricksize(64, 64, 64)
.datatype(SampleDataType::int16)
.datarange(-32768,+32767);
ZgyWriterArgs secondargs = ZgyWriterArgs()
.filename(filename)
.ilstart(1).ilinc(2)
.xlstart(500).xlinc(5)
.zstart(100).zinc(4)
.hunit(UnitDimension::length, "m", 1)
.zunit(UnitDimension::time, "ms", 1000)
.corners(ZgyWriterArgs::corners_t{5,7,5,107,205,7,205,107});
// Create with basic information only.
std::shared_ptr<OpenZGY::IZgyWriter> writer =
OpenZGY::IZgyWriter::open(firstargs);
writer->finalize(std::vector<OpenZGY::DecimationType>{}, nullptr);
writer->close();
writer.reset();
// Try to re-open it. Should work since there are no data blocks.
writer = IZgyWriter::reopen(secondargs);
// Continue as in test_write(). Probably should have refactored to share code.
std::vector<float> data(2*3*4, -1000);
const OpenZGY::IZgyWriter::size3i_t origin{0,0,0};
const OpenZGY::IZgyWriter::size3i_t bsize{64,64,64};
const OpenZGY::IZgyWriter::size3i_t count{2,3,4};
float fortytwo{42};
writer->writeconst(origin, bsize, &fortytwo);
writer->write(origin, count, data.data());
writer->finalize(std::vector<OpenZGY::DecimationType>{}, nullptr);
writer->close();
}
void test_write()
{
LocalFileAutoDelete lad("testfile.zgy");
do_write_once(lad.name());
do_check_written(lad.name());
}
void test_update()
{
LocalFileAutoDelete lad("updatelocal.zgy");
do_write_twice(lad.name());
do_check_written(lad.name());
}
// TODO-TEST:
//void test_update_cloud()
//{
// CloudFileAutoDelete lad("updatecloud.zgy");
// do_update(lad.name());
//}
void test_historange()
{
LocalFileAutoDelete lad("testhisto.zgy");
......@@ -1279,6 +1367,8 @@ public:
register_test("api.readbadvt", test_readbadvt);
register_test("api.readbadpos", test_readbadpos);
register_test("api.write", test_write);
register_test("api.update", test_update);
// TODO-TEST: register_test("api.update_cloud", test_update_cloud);
register_test("api.historange", test_historange);
register_test("api.lod_lowpass", test_lod_lowpass);
register_test("api.lod_weighted", test_lod_weighted);
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment