Zimbra OSE backup script
2010-06-13
From now on this backup script is part of oSource Development and thus will not be updated here. See oSource Development for further details. Visit oSource Development here…
This script will do various backups of Zimbra depending on which you choose, most of them are cold backups except the message backup which hot copies the store folder for possible individual mail retrieval later on.
Due to the size of the script it is attached as a file to this post.
There are some options in the script that should be changed before running it, below are these rows with a short explanation on what they are for. The ZimInstPath should point to the installation directory of Zimbra excluding the Zimbra folder. The ZimHome should say Zimbra unless you changed default home folder of Zimbra. Finally the two last variables ZimBackupPath & ZimTempPath should point to where you want this script to put the backup files, these can be separated from each other if it is needed but i recommend not to do it.
Important: As being pointed out on some systems the rsync isn’t installed by default and rsync is highly used in the script and thus needs to be installed.
Download here: ZimBackup.sh
2008-06-10:
Help inside the script for configuration options.
Logging capability, now you can log your backup either with minimal information or in a verbose mode that throws pretty much anything into the log useful for when trying to find backup problems.
2008-04-18:
Script has been updated with new functionality that enables file transfer with ftp & scp protocols. Following changeable options has been added to the script, ZimFtpEnable = Enable ftp file transfer, ZimScpEnable = Enable scp file transfer, ZimFtpOpt = Extra options for the standard ftp command, ZimScpOpt = Extra options for the scp command, ZimFilehostUser, = Username for connection, ZimFilehostPass = Password for connection, ZimFilehostAddress = Address to the host that should receive the files, ZimFilehostFolder = Folder on host side where file should be stored, ZimDeleteLocalFile = When file has been transfered delete the file. All of these can be found under the section File Transfer Settings and most of these settings are for common for all file transfer protocols.
2008-04-19:
Fixed a ‘expect’ quirk regarding timeouts that broke file transfers taking more than 10 seconds to complete.
2008-05-18:
Now it is possible to decide how many local copies of backup files to keep based on file timestamps. Lets say that we do backup once a night at 12 and want to keep to 2 successive copies locally and delete the rest, the by setting ZimDeleteTimeSet to 3000, which is 2880min which is equal to 2 days + 120min as a margin in case the backup should take a little longer than usual, we will keep the number of files we want locally. The time is set in minutes to ensure some degree of flexibility.
August 22nd, 2008 at 8:26 pm
great work, thanks
can you add some words about restoring diff backups?
August 24th, 2008 at 7:50 pm
Thanks.
I will write a how-to on a restore in a while, but for now you can use one from Zimbra found here: http://wiki.zimbra.com...
September 18th, 2008 at 9:49 pm
Soulskater,
Thanks a lot, if you need some help with the restore script please ask, i’ll be glad to help.
[]‘s
Guto
November 3rd, 2008 at 2:22 am
[...] we used "A Simple Shell Script Method" from wikii and now we are using scripts from http://www.swedcore.net Blog Archive Zimbra OSE backup script. This server is assigned 1GB of RAM and even though we have lots of users they are not heavy [...]
November 4th, 2008 at 3:43 am
During the script execution Mail Server will still be running continuously and able to send/receive emails?
November 4th, 2008 at 1:30 pm
No, all services is shutdown during backup, i don’t know a way around it sorry, but i have tried to limit the time that it is kept offline as much as i can, by syncing backup folder with system folder trying to copy as much as i can before it is shutdown. If anyone know how to allow the server to receive mail during backup let me know and i will test and implement.
By the way thanks Guto, i will keep it in mind.
November 4th, 2008 at 3:23 pm
Hi soulskater,
Does your script automatically runs again the zimbra process?
What is the approximate down time ?
I have to run a zimbra server for 25 users, It’s not so much but it’s critical for the business the uptime of the mail server.
November 5th, 2008 at 10:25 am
Yes my script start Zimbra again during backup. The downtime for my server when i do a full backup is about 2-3 minutes.
November 5th, 2008 at 11:11 am
A non-downtime solution probably would go with a paid version of Zimbra, or do you know any other script/solution without this downtime limitation?
November 5th, 2008 at 12:11 pm
No sadly i don’t, as you said this function is probably limited to the paid version of Zimbra.
November 5th, 2008 at 3:20 pm
One thing you should keep in mind is that most mail servers has so called mail queues functionality that keeps the message for a while if the first attempt of sending fails it keeps it for a while and then tries to resend it, so you should not loose any mail coming in during this short downtime.
November 6th, 2008 at 8:41 pm
Thanks!
I will keep an eye on your blog by time to time
November 7th, 2008 at 9:20 pm
Do so, I appreciate it.
November 19th, 2008 at 8:59 pm
Thank you
I am the Brazilian, and am curious to find out if your script works on Debian Etch 4.0a (version October 2008).
If you can tell me.
I am implementing the university within the Tool juniper.
Thank
Fabiano
November 20th, 2008 at 10:21 pm
Thank you very much for this. Reading the code it appears the diff backup generates a fully intact backup file, so all you need is that individual file generated from the diff backup to do a restore? It’s diff in the sense that only files that have changed get copied (via using rsync), but the actual backup file produced is a complete backup. Is that correct?
Second, the link you gave above on August 24 for restores doesn’t seem to work. Would you mind posting it again? I imagine it is pretty straightforward as the script produces a full binary copy of the zimbra directory.
Thanks again.
November 21st, 2008 at 3:21 pm
Fabiano:
I cannot honest say if it does or not, the only way to know for sure is to try it, but it shouldn’t be a problem since i haven’t used any OS specific tools in the script.
rwa:
You are right in the assumption that the diff produces a full backup to, the different in full and diff is that in diff it only copies the changed files from the Zimbra to the already existing backup folder, while full cleans out the backup folder and does a clean copy every time, but they both produces a full backup but the downtime for Zimbra is shorter with the diff, I’m currently working on making the diff contain only the changed files thus reducing the file size for this backup type.
Links for restoring Zimbra: Link 1, Link 2
January 14th, 2009 at 12:53 pm
Hello,
I tried to use ftp option to transfer the zimbra backup but i got this error message:
expect command is missing, this is required for file transfer options, script exiting…
Thanks.
January 16th, 2009 at 9:38 am
Hi Pascal
First, thanks for using my script, secondly, this message is due to that the script uses a command called expect to send/receive messages from/to external commands like ftp, this is something that isn’t installed by default on any system and must be installed separately, if you open up the script in a text editor you should see this information at the top of the script.
Hope you get it to work and will be pleased with it.
January 16th, 2009 at 1:16 pm
Hello,
Thanks for your answer. FTP client had installed and is working on zimbra host. FTPD server installed too on destination host so what is wrong. I had tried with ssh but it’s the same problem.
Thanks.
January 16th, 2009 at 1:20 pm
May i have to install expect tcl 8.4 package perhaps ?
January 19th, 2009 at 9:55 am
Hi.
That’s correct you need the expect package for the file transfer functions.
January 20th, 2009 at 8:17 pm
Howdy! Mucho thanx for sharing your hard work and making things easy for lazy admins like myself!
I do have a concern:
zimbackup.sh: line 213: rsync: command not found
Now, I know what rsync is, and I know I don’t have it installed. The problem is this: why is the rsync command being invoked? I didn’t see any documentation about rsync.
January 20th, 2009 at 8:18 pm
Oh, I am truly lazy! I decided to visit line 213 after I made my comment.
“# Hot sync of mailbox messages”
So I guess I need rsync installed? As far as I know, I got a good hot backup of the messages…
January 20th, 2009 at 8:20 pm
Geeze, I really need to calm down before I post comments — I do apologize
I installed rsync and now the hot backup of the message store is MUCH bigger this time. Duh.
Sorry again for so many comments; I should know better.
January 22nd, 2009 at 10:19 am
Hi.
Hmm, isn’t rsync installed on your system by default???
As far as i can remember it is on my system(Ubuntu 8.04 Server). Maybe i should make a note about that in the text above.
Thanks for bringing this to my attention Matthew.
January 27th, 2009 at 11:26 pm
Hi,
like all the posts above, muchos gracias for this awsome script. i used to shutdown zimbra tar the whole bunch and start it up again.. this resulted in much downtime. Your script reduces the downtime to about 1,5 minutes..
As a feature request, what would be great imho to lower the rsync time for offsite backups. now when i rsync offsite, it takes about 5 hours.. that sucks.. and the store is growing faster and faster lately.
Do you have any plans on implementing such feature in the future ?
thanks again…
greets
Mayk
February 1st, 2009 at 6:09 pm
Great work! really nice script
February 2nd, 2009 at 2:28 pm
Hi.
Thanks Sergei.
February 2nd, 2009 at 8:11 pm
hi, soulskater
I am again Fabiano from Brazil, got success with the juniper and also succeed in creating the BACKUP SCRIPT with not only know how to restore the file it creates. give, says the file is encrypted and was not informed about the key as could be through with this issue, which must extract the contents of this file.
agredeço help.
I am only in part.
sorry for the English, I am still a little using the Google Translate, or even understand all the English, soon will be more fluent.
Fabian
February 8th, 2009 at 5:48 pm
Hi Fabiano
The file should not be encrypted, it should work to just untar it like this: tar xvfz filename.tar.gz
Just remember to put it a folder of itself so it doesn’t gets unpacked with other files.
/soulskater
February 13th, 2009 at 5:19 pm
Hello Soulskater!
Has I told previously, I’ll keep an eye in your blog. Few lame questions regarding your script:
- What are your advices regarding the cron backup setup? Like, for daily: msgdiff, weekly: msgfull, monthly: full ?
- What do you think about receive a small EMAIL log for the executed backup?
- How about the RESTORE procedures? I think it’s important to include in the script header.
It’s all for now!
Thanks!
February 15th, 2009 at 6:03 pm
Hi.
Regarding cron setup i have done as follows to minimize disruption with other zimbra cron jobs:
0 3 * * * /opt/scripts/ZimBackup.sh full > /dev/null 2>&1
0 12 * * * /opt/scripts/ZimBackup.sh msgfull > /dev/null 2>&1
I have been thinking about some sort of e-mail log but have not looked into it any further for the moment.
When it comes to restore it is a bit more complicated to utilize a script for it, or do you mean a how-to that describes on how to restore?
/soulskater
February 16th, 2009 at 4:49 pm
Hi,
Yes, I mean a “quick how-to” so the user can know how to restore the backup files. I think it’s important even though if it’s obvious for many people.
February 17th, 2009 at 3:29 pm
Mayk:
thanks for your reply, i missed your reply somehow, not my intention. The rsync time depends on a number of factors, for example, network speed between your computers, the amount of data it has to transfer, protocol used for the sync, like ftp, ssh a.s.o unfortunate there isn’t anything i can do anything about those issues.
Andre:
I have created a draft for a little How-To about restoring so keep checking back and hopefully i will get it done soon.
February 23rd, 2009 at 1:10 pm
Hello,
0 3 * * * /opt/scripts/ZimBackup.sh full > /dev/null 2>&1
0 12 * * * /opt/scripts/ZimBackup.sh msgfull > /dev/null 2>&1
Is there any reason you’re not using diff backup?
February 23rd, 2009 at 6:03 pm
Hi, first of all thanks for the script.
Can you clarify connection between your script and this one?
(http://www.osoffice.de/)
Cause, as far as I understand, at the zimbra wiki page there is link for your script and listing of another script.
(http://wiki.zimbra.com/index.php?title=Open_Source_Edition_Backup_Procedure#Backup_Shell_Script_with_Compressed_.26_Encrypted_Archives)
1) Are this scripts connected somehow or not?
2) If someone have a backUp, using your script in fullmsg mode, and clear zimbra server with no user date, it would be impossible to restore date, cause there is no mysql data. Or I missing something?
I’m a new zimbra user and just thinking about date backup.
Thanks in advance.
February 26th, 2009 at 10:59 am
Hi iliya
1. I think that the link just has been missplaced and should be above the title “Backup Shell Script with Compressed & Encrypted Archives” they are not connected to each other.
2. If you have a backup with my script you should still have the original date on the files residing in the backup and therefor should not have any problem restoring individual mail using “zmlmtpinject” but that needs the user(s) to be recreated before you are able to do that, and sadly if i remember this correctly the dates on the mail injected will changes to the date of injection date, at least that was the behavior before with “zmlmtpinject”.
Hope this answers your questions.
February 26th, 2009 at 11:01 am
Andre:
The reason is that i have no need for it, i doesn’t have much data on my server so a full backup for me works good.
March 11th, 2009 at 6:32 pm
I’m can see how easy this makes creating backups, and now i’m looking for a way to restore this backups with all this different options. On a test server I have done a full backup then i reinstalled the Server with the same specs, names ip and so on, also reinstalled zimbra with the same name, once that was done i stoped the server, was restored the backup and reset the rights, everything was working great. Thanks, but I’m not sure how to restore a Differential backup, or a messages only backup. Can anyone point out some documentation on that?
April 5th, 2009 at 5:40 am
Here’s an idea: consider adding Amazon S3 support. I know, I know, feature bloat! But this is a highly popular backup destination and would make a great alternative to FTP and SCP.
April 5th, 2009 at 2:11 pm
Rodrigo:
When restoring a diff backup when been using my script is done exactly as with a full backup, my script copies only changed files, and if the tmp folder under backup dir is removed it does a full backup. Message backup is more hassle to restore, you cannot just restore the messages into it’s folders since the are removed from Zimbra database when deleted in the gui, you must use a tool called zmlmtpinject(part of Zimbra) to re-inject individual mail into Zimbra, when restoring them from packed file place them outside of Zimbra folder before using zmlmtpinject.
Matthew:
Amazon S3? hmm actually never heard of it, might be interesting tho.
/soulskater
April 9th, 2009 at 3:43 am
Hi, I do a full backup on weekend, and on the week days I do a full msg. backup.
This is ok for restore the server ??
I mean, I can restore the full backup (using “Rodrigos post”) and then once restored the the mail server restore (update) the full messages backup?
Thanks for your time.
Andres
April 22nd, 2009 at 7:35 pm
yours script is great and kudos to you and your effort.Everthing works fine for me except the remote ftp part.I have all the details for the remote ftp host…password and user name.I could manually ftp to the
remote host from the server but can’t ftp the backup files using the script.could you please guide me on this,your help will be appreciated.
May 6th, 2009 at 5:35 am
I did a full system backup of the server today. The server is back up and running now and email messages are being sent all seems well except when I go to log in with the admin console it wont let me in. Also when I log in to the webmail with my email account it logs me out as soon as I log in. Any help would be greatly appreciated.
Thanks Liam.
May 11th, 2009 at 8:47 pm
I’m Sorry for not answering your questions, been sick for a while now.
agd, as stated in my earlier post, message restore is a bit complicated, you have to use a tool in Zimbra called zmlmtpinject, the downside of this is that all messages will get the date when they are restored, the msg backup should be used foremost as an option to restore individual mail to a specific user.
tashi, if you enable logging in the script and try a msg backup just to test and look on what it says, i think you will se why it doesn’t work, you can post the error message here if you need further help.
Liam, do you get any sort of error message? i know that sometime when i had to restore a backup the certificates for my zimbra got broken and i had to regenerate new, don’t remember how but i know that i found it in the zimbra forum, let me now if you need more assistance.
Thanks to you all for using my script.
/Marcus
September 19th, 2009 at 9:20 pm
Hi, It’s possible to use the script to backup on version ZCS 6.x OSE ??
Thanks
September 19th, 2009 at 11:47 pm
Hi.
I’m not sure haven’t upgraded to version 6 yet. Will do soon and will test the script functionality against v. 6.
/Marcus
October 1st, 2009 at 12:07 am
As it looks now the script will work with version 6 of Zimbra.
/Marcus
October 1st, 2009 at 8:46 pm
Great news. Thanks!!
October 23rd, 2009 at 4:34 am
This is great! Now I want to see your ways for us readers to become more involved! Expect an email later today.
November 17th, 2009 at 1:29 am
Hello Soulseeker,
3 questions:
1# Why my BackupZimbra.log is always 0 bytes?
2# /tmp seems never to be removed.
3# Any plans for a script update?
I probably shouldn’t have this but… script is executed by root.
Cheers,
AT
November 17th, 2009 at 8:35 am
Hi.
1# Have you enabled the log function in the script?
2# The tmp folder is never removed since it contains backup information vital to the diff backup options. if it is a feature wish that it will be possible to remove this folder i can implement it.
3# Yes i have plans for script update, just got stuck with other stuff.
/Marcus
November 17th, 2009 at 12:24 pm
Hello.
1#
Log was disabled, although the script creates the file anyway (but it does not populate it) which lead me to think it was enabled. Maybe you could take a look at this in the next releases?
2#
tmp/ folder could be removed after a successfully tgz. This is date duplication. My Backup is 5GB and tmp folder is 8.9GB, which is not good since I’m struggling for disk space.
Well… I can have an automated crontab to delete tmp for now, but I would be nice in a next future release if It would delete it automatically.
3#
I look forward to see it! What are the feature highlights?
November 23rd, 2009 at 12:44 am
I just realised my diff backups are taking 18min and the full backups 30min!
March 9th, 2010 at 9:50 am
Hi to all !
Is there some progress in development of diff option ? I have noticed, that diff backup is same size as my full backup, and that is for me problem, because, as I have calculated, my diff daily is not more that 50 to 70MB, and my dedicated space on backup server is for 4 full backups + 24 * diff on the month.
By that diff command is not working for me what I have planned …
Thx for help … !
March 9th, 2010 at 9:39 pm
Hi,
There is development of the script even tho it is in a slow phase right now.
The diff backup is for the moment a kinda ugly diff backup, what it does is to compare a backup version with the live and then copies the changed files to the backup version still making it a full backup, it is in an overhaul to only contain the diff files and making it much smaller.
June 20th, 2010 at 1:34 pm
Just want to says thanks for all your comments here regarding this script. They have been invaluable to me and the shaping of this script.
I will still develop this script but on a new site called oSource Development and hope you will rejoin me there and continue to support this script.
Please visit oSource Development for further information regarding this script. There is a new version available over there for you to download and use with a lot of improvements.
http://dev.osource.se/
/Marcus Uddenhed