At Zumasys, we are passionate about MultiValue databases such as D3, jBASE and OpenQM. The applications that rely on these databases are often mission critical to the business. More often these days our customers are turning to a wonderful solution called D3 Hot Backup as an effective way to keep their D3 systems replicating to a second server over a local or wide-area network.
Hot-Backup is a feature of D3 that allows the server to record transactional data (writes, updates, deletes, file creation and deletion) and send these transactions to a secondary server which will receive the transactions and when the customer originally called, their HOT-BACKUP stopped running and their Hot-Backup transaction queue filled to configured maximum capacity and once this happens, the Hot-Backup service stopped enqueing transactions. At this point the Hot-Backup feature is rendered useless and until the reason is corrected, Hot-Backup is non-functional!
I attempted to restart the hot-backup server on the primary (master) server, it would start ok. (Great! I thought…)
Starting the hot-backup on the secondary (slave) server, the primary (master) server would try to send transactions from the transaction queue to the secondary server and it would fail with a: “SEGMENTATION VIOLATION, ABORT” message and the hot-backup server would stop. (argh!) Any attempts to restart would result in the same error.
The steps that were taken: (these are the normal steps you take when syncing hot-backup primary/secondary servers)
- Cleared transaction queue on the primary server after stopping transaction enqueing.
- Restarted transaction queueing before performing a full file-save on the primary server. This would allow changes to be applied once the secondary server was restored and hot-backup on that server was started.
- Transferred the pseudo file-save over to the secondary server.
- Performed a full restore on the secondary server.
- Made sure that the Secondary server when coming back up did not start the hot-backup process. (there is a prompt at coldstart)
- After Secondary server was back up, changed the System ID (ed messages config, and change to the correct system id)
- Started hot-backup on the secondary server which started correctly and remained running.
- Started hot-backup server on primary (master) server
…and the same “SEGMENTATION VIOLATION, ABORT” message appeared and the master server stopped once again.
After much diagnosing and working with Rocket Software, we seemed to have identified the issue and was instructed to do steps 1-8 again that evening.
Tried it again and we ended up with the same results! (Argh!) Called Rocket Software again on same ticket and talked to the same technician once again. After some discussion on theory regarding hot-backup and features/functions, we found the following…..
D3 has a file-of-files that is basically the driving force in all things relating to files/accounts and auditing in the D3 server. It gets updated every time a File-Save occurs. This file-of-files contains each file on the system along with auditing information. (Clear-File, etc. storing data/time port/user, etc.)
The main issue was an item in the file-of-files had an extremely large attribute (900,000+ bytes) which was causing the SEGMENTATION VIOLATION ABORT to occur. There is some internal limitation and once it reached this limitation, the function D3 uses to work with this type of item fails.
In order to correct this issue, we cleared up attributes 18-21 (actually cleared the data in these attributes) and then set attribute 17 to a “C” which causes the file to not collect any auditing information.
There is a command in D3 that you can use to turn off auditing on the D3 server called: “LOG-CLEAR-FILE-OFF”
This command will stop D3 from collecting auditing information and add any data into these attributes in the file-of-files items.
Most customers that I have ever dealt with in the D3 world has never used or even knew about auditing on their D3 server.
NOTE: When customers call and cannot explain why their D3 server is using up disk space and cannot understand why. Little by little these items in the file-of-files file is continually growing for years! So best bet on most D3 system is to install the “LOG-CLEAR-FILE-OFF” in the user-coldstart or system-coldstart procs, this way it will globally be turned off and stop accumulating additional data that is effectively useless to the customer.
One last tidbit of information that might or might not be useful in your daily activities in the D3 world…
Checking the size of any item in a file is just a matter of typing “LIST FILENAME *A9999” which will produce a listing of all items in the file and their actual size. This is a great tool to help determine which items are taking up large amount frames on the disk.
So by typing “SORT FILE-OF-FILES BY-DSND *A9999 NAME *A9999” will show me the largest to the smallest items in the file-of-files file which will alert you to the problem of finding those pesky “SEGMENTATION VIOLATION” offenders!