# Synced Append
A library that can be used to append multiple text files and do commits so that all files are in sync and doesn't leave in corrupted states between process crashes and restarts. In case you only have just one file to append still this library can be useful as it ensures files will not stuck in corrupted states.
This supports Atomicity, Consistency and Durability in ACID. Isolation is not supported.
Note: It's assumed fs.fsyncSync() ensures data written to the hardware.
The Problem
If strings are appended to files using write() it's not guaranteed data written to the disk in an atomic manner. This makes sense as if it initiates a write to disk for each small string append, OS will be spending lot more time doing disk IO. But the problem with this is if the process crashes in an unexpected manner (e.g. a SIGKILL), it's possible that all the appends you did might not reach the disk.
Solution
Please go through following links to understand basic concepts on how to solve these types of problems.
How to get it
npm install synced-append
Example
SyncedAppend = require 'synced-append' files = country: './data/country.csv' city: './data/city.csv' # - First parameters is the file used to store states so that commits can be # rolledback. # - Second parameter is the set of files that need to be in sync # - At the time of object creatation any previous uncomiited work will be # rolled back. sync = './data/rollback.log'files # starts the sync for appends. This creates the rollback log syncstart countryWriter = syncgetFile 'country' cityWriter = syncgetFile 'city' # append strigns to files countryWriterappend 'code,name\n' countryWriterappend 'LKA,Sri Lanka\n' cityWriterappend 'id,name,countryCode\n' cityWriterappend '0,Colombo,LKA\n' cityWriterappend '1,Kandy,LKA\n' # Commit above appends to files. This doesn't leave corrupted states in case of # crash syncsyncStop newFiles = language: path: './data/language.csv' encoding: 'utf8' # Since sync was stopped previously can add new files when starting again syncstart newFiles langWriter = syncgetFile 'language' countryWriterappend 'USA,United States of America\n' langWriterappend 'countryCode,lanague\n' langWriterappend 'USA,English\n' # Commits above changes and auto start. # This is euqauvlent to sync.syncStop and sync.start() syncsync
Example 2
# Run this program multiple times. Every time it runs it writes to a # new log file. SyncedAppend = require 'synced-append' # recovers last written file sync = "./data/example2_rollback.log" logWriter = syncgetFile 'log' if logWriter? consolelog "Recovered the file at " # Every time we write to a new file files = log: "./data/log-.txt" # starting write to the new file syncstart files if not logWriter? logWriter = syncgetFile 'log' logWriterappend "File written at " # Let's commit the append syncsyncStop # File paths can be changed on the file when sync is stopped files.log += ".more" syncstart files # Try removing these two lines. If no bytes are committed to a file # the file will be removed when recovered. logWriterappend 'first line\n' syncsync # Force stop the program with uncommitted appends = consolelog """ exit \n Process is force stopped. There can be uncommited appends in the file """ processexit code setTimeout exit1001 # Append the file without commiting so that file left with uncommited # appends when the program exits MAXN = 1<<20 = -> # Make sure appends are large enough to fill the file buffer so that # actual file write will happen. for i in 0...MAXN logWriterappend "" setTimeout next0 next
License
MIT