LiveAverage » Coding

Managing backup retention… with one line of Powershell

JR — Wed, 30 Nov 2011 18:09:08 +0000

Ok, I used four lines, but my needs for retention might be a bit more complex than most. I also spaced each pipeline command, so it looks longer than it should, but readability is important. Additionally, there’s a good half-page of comments in the script than can safely be ignored, unless you want to know what was going through my mind. Most of these related directly to my desired retention periods.

For testing purposes, the last two “lines” only print out the listing of files that would be deleted.

Param ($backupDir)

######
# Handle 30-day backup recyclables first (no directory recursion since each backup is in its own directory)
# The current backup schedule performs weekly configuration backups with 1-month retention.
######

Get-ChildItem "$backupDir\backup_data" | `
`
Where-Object {(($_.FullName -match "configuration_isxpr01_") -and ( $_.CreationTime -lt ((Get-Date).AddMonths(-1)) ) )} | `
`
ForEach-Object {`
                    Out-File -filepath "$backupDir\log" -inputobject $((Get-Date -format g) + "Removing " + $_.FullName + " from " + $_.CreationTime) -append`
                    Remove-Item $_.FullName -recurse}

######
# Handle 7-day backup recyclables next (no directory recursion since each backup is in its own directory)
# The current backup schedule performs weekly configuration backups with 1-week retention.
# No "overwrite option available, so this script manages removal
######

Get-ChildItem "$backupDir\backup_data" | `
`
Where-Object {(($_.FullName -match "backup_isxpr01_") -and ( $_.CreationTime -lt ((Get-Date).AddDays(-7)) ) )} | `
`
ForEach-Object {(Remove-Item $_.FullName -recurse)}

######
# Handle daily trend data recyclables next (use recursion parameter).
# The current daily trend export schedule requires 1-month retention for 24-hr trend data.
######

Get-ChildItem -include *.zip -recurse -path "$backupDir\trend_data_export" | `
`
Where-Object {(($_.FullName -match "24-hr") -and ( $_.CreationTime -lt ((Get-Date).AddMonths(-1)) ) )} | `
`
ForEach-Object {($_.FullName)}

######
# Handle monthly trend data recyclables next (use recursion parameter).
# The current monthly trend export schedule requires 5-year retention for 30-day trend data.
######

Get-ChildItem -include *.zip -recurse -path "$backupDir\trend_data_export" | `
`
Where-Object {(($_.FullName -match "30-day") -and ( $_.CreationTime -lt ((Get-Date).AddYears(-5)) ) )} | `
`
ForEach-Object {($_.FullName)}

OpenFiler errors that only I seemed to experience…

JR — Tue, 26 Jul 2011 16:51:03 +0000

Well, I’ve finally deployed some production Openfiler ESA 2.99.1 machines as home-brew iSCSI boxes, primarily used for backups or low-stress virtual storage. Yes, they’re great — my basic write speeds on a Core 2 Duo box (recycled Dell Precision 390 workstation with 2GB of RAM and a single 1TB drive *no* RAID):

administrator@mail:/backup-iscsi$ sudo dd if=/dev/zero of=garbage bs=131072 count=20000
20000+0 records in
20000+0 records out
2621440000 bytes (2.6 GB) copied, 40.8493 s, 64.2 MB/s

I’m pretty happy with that. What I’m not please with is a 56MB log (/var/log/secure) filled with strange messages:

Jul 26 12:25:46 e0-002 sudo: openfiler : TTY=unknown ; PWD=/opt/openfiler/var/www/htdocs ; USER=root ; COMMAND=/usr/bin/uptime
Jul 26 12:25:46 e0-002 sudo: PAM unable to dlopen(/lib64/security/pam_gnome_keyring.so)
Jul 26 12:25:46 e0-002 sudo: PAM [error: /lib64/security/pam_gnome_keyring.so: cannot open shared object file: No such file or directory]
Jul 26 12:25:46 e0-002 sudo: PAM adding faulty module: /lib64/security/pam_gnome_keyring.so

Okay, so the first message isn’t strange. In fact, it’s normal. But I couldn’t figure out why a non-GUI (yes, it has a web interface, but no default window manager) distro was trying to load the Gnome keyring shared object. I still don’t know why. What I do know is how to get rid of this message:

Verify which PAM files reference this shared object:

[root@e0-002 pam.d]# cd /etc/pam.d/
[root@e0-002 pam.d]# grep -i -n 'pam_gnome_keyring.so' *
system-auth:5:# FL: Have (patched) pam_gnome_keyring.so grab the password before pam_unix.so
system-auth:6:auth        optional      pam_gnome_keyring.so
system-auth:16:password    optional      pam_gnome_keyring.so

In this instance, the ‘system-auth’ file contained these troublesome pam_gnome_keyring.so entries.
Fire-up your favorite text editor and comment those lines.
Reboot & enjoy a log without an absurd amount of irrelevant errors.

I found this referenced once on the OpenFiler forums, but no answer or explanation. I hope this helps someone else that might experience a similar problem.

I’ll also be doing a write-up related to OpenFiler volume backups with LVM snapshots.

Add ‘Submitted Tickets’ Listing Page for Joomla! RSTickets

JR — Fri, 10 Jul 2009 17:15:21 +0000

If you haven’t heard, RSTickets! is an advanced Joomla! Help Desk ticketing system that allows you (or a team of yous) to manage and keep track of your clients’ issues. It’s actually one of the few effective, useful Help Desk systems available for the Joomla! 1.5+ framework that I would personally recommend. Unfortunately, it’s still under development and lacks certain features that one may desire, such as a read-only listing page that displays tickets already submitted to you or your department.

My Problem:

I noticed internal network clients were submitting several duplicate tickets related to a shared problem (i.e. printer trouble, network outages, etc.). I couldn’t blame them since they couldn’t view previously submitted tickets, so I decided to create a quick + dirty page that pulls “open,” or “on-hold,” tickets from a specific department’s table of submitted [active] tickets and displays them on a Joomla! article page (using the Sourcerer plug-in to execute custom PHP with {source} tags). Pasting this code into an article without the Sourcerer plug-in [or some sort of plug-in for executing PHP] will do nothing. Also note the include file for making a raw connection to your MySQL database. This is required (and should be stored in a directory with the appropriate permissions to prevent outside read access).

If you’d like to simply list the number (amount) of tickets (open, closed, on-hold) for all departments, you might want to check the unreleased version of RSJoomla!’s RSTicket Module.

{source}
';

echo 'Please check the open, pending, or on-hold tickets listed below before submitting a support ticket. The list below can be utilized as an informal gauge for IT response times. If a duplicate support ticket is submitted, you may cancel it yourself or it will be deleted by the IT Department upon review. Thank you for your cooperation.';

echo 'SUBMITTED SUPPORT TICKETS:';

echo 'Submitted:Username:Ticket Code:Subject:Status:Priority:';

while($row = mysql_fetch_array($result2))

{

$custid =  $row['CustomerId'];
$userquery= mysql_query("SELECT name FROM jos_users WHERE id=$custid") or die(mysql_error());

$username = mysql_fetch_array($userquery);

echo '' . date("Y-m-d H:m", $row['TicketTime']) .  '' . $username['name'] . '' . $row['TicketCode'] . '' . $row['TicketSubject'] . '' . strtoupper($row['TicketStatus']) . '';

if ($row['TicketPriority']=='high'){

echo '';

} else if ($row['TicketPriority']=='normal'){

echo '';

} else if ($row['TicketPriority']=='low'){

echo '';}

echo strtoupper($row['TicketPriority']) . '';

}

echo '
SUBMIT A NEW SUPPORT TICKET';

echo '
';

?>
{/source}

connect_custom.php :

Altermime, Postfix/Zimbra, and Headaches

JR — Wed, 24 Jun 2009 19:43:11 +0000

EDIT: I have since removed altermime after installing a MailScanner spam relay for our Zimbra server to use. Because, by default, MailScanner appends a default signature to all outbound email, it was very simple to modify the signature rules to accomodate our mandatory disclaimers for different domains.

I had the pleasure of applying mandatory disclaimers to all [outbound] emails at my workplace today… ~Joy~ … I had the assumption it’d be rather easy, but Altermime and Postfix were a bit finicky to work with. After editing the master.cf I ended up customizing my own ‘disclaimer’ shell script.

#!/bin/sh
INSPECT_DIR=/var/spool/filter
SENDMAIL=/opt/zimbra/postfix/sbin/sendmail

# Exit codes from
EX_TEMPFAIL=75
EX_UNAVAILABLE=69

# Clean up when done or when aborting.
trap "rm -f in.$$" 0 1 2 3 15

# Start processing.
cd $INSPECT_DIR || { echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; }

cat > in.$$ || { echo Cannot save mail to file; exit $EX_TEMPFAIL; }

# Verify this mail is not incoming or internal-only
# We don't need disclaimers for either one of these cases.

# Debug:
#echo "output: $from_address" >> /tmp/tempoutput.txt

#Grab the from address:

from_address=`grep -m 1 "From:" in.$$ | cut -d "<" -f 2 | cut -d ">" -f 1`

#Verify whether your domain is in the from address.
#If it is, proceed to distinguish WHICH domain is sending outgoing mail and tag it appropriately:
#If not, then that would be incoming mail, so leave it alone:

#Additional (else if) conditional checks will be added to determine if the email is inner-office comm:
# to_address= `grep -m 1 "To:" in.$$ | cut -d "<" -f 2 | cut -d ">" -f 1`
# if [[ $from_address == *domain* && to_address == *domain* ]]; then ....
#
# This additional condition requires more debugging...

if [[ $from_address == *domain* ]]; then

# Debug:
#echo
#echo "FROM: $from_address" >> /tmp/tempoutput.txt
#echo "THIS GETS ALTERMIMED" >> /tmp/tempoutput.txt
#echo

#Check to see which domain is sending outgoing email,
#then tag it with the appropriate disclaimer:

if [[ $from_address == *subdomain.domain* ]]; then

# Debug		echo "THIS GETS APDD" >> /tmp/tempoutput.txt
/usr/bin/altermime --input=in.$$ \
--disclaimer=/opt/zimbra/postfix/conf/disclaimers/apd-disclaimer.txt \
--disclaimer-html=/opt/zimbra/postfix/conf/disclaimers/apd-disclaimer.txt \
--xheader="X-Public-Record:" || { echo Message content rejected; exit $EX_UNAVAILABLE; }
else
# Debug		echo "THIS GETS COAD" >> /tmp/tempoutput.txt
/usr/bin/altermime --input=in.$$ \
--disclaimer=/opt/zimbra/postfix/conf/disclaimers/coa-disclaimer.txt \
--disclaimer-html=/opt/zimbra/postfix/conf/disclaimers/coa-disclaimer.txt \
--xheader="X-Public-Record:" || { echo Message content rejected; exit $EX_UNAVAILABLE; }

fi
fi

# Might need to remove -i switch for truncation problems depending on your MTA version...

$SENDMAIL -i "$@" < in.$$

exit $?

Bashing MySQL Dumps

JR — Tue, 29 Jul 2008 14:45:14 +0000

A quick set of batch scripts I wrote up (two of the three scripts, anyway) for dumping all of my (specified) MySQL databases into an archive for backup. The log mailing (emailsql.pl)requires Perl and the MIME:Lite module to correctly function. I’ve also utilized a wrapper script so the log outputs to a separate file [to be mailed]. There’s also a ‘dummy’ log file I use in my crontab file, though this isn’t really necessary:

mysqlbackupwrapper.sh

#!/bin/bash
#Wrapper script used to call the primary backup script and output to a specified file
sh /home/administrator/scripts/mysqlbackups > /home/administrator/scripts/sql.backup.log 2>&1

mysqlbackup.sh:

#!/bin/sh
#Timestamp for your logs:
date

#Dump the databases - Make sure to specify your root or user password following the -p switch:
mysqldump -uroot -p --opt intraforum > /home/administrator/scripts/sqldata/intra_apdforum.sql
mysqldump -uroot -p --opt joomla_intranet > /home/administrator/scripts/sqldata/intra_intranet.sql
mysqldump -uroot -p --opt mysql > /home/administrator/scripts/sqldata/intra_mysql.sql
tar -zcvf /home/administrator/scripts/sqldata.tgz /home/administrator/scripts/sqldata/*.sql
echo
echo "Backup completed successfully for: "
echo
echo "MySQL - PHPbb3 Forum"
echo "MySQL - Joomla 1.0.X Intranet"
echo "MySQL - Intranet MySQL Tables"
echo
echo "Copying to SERVER.yourdomain.local . . . ."
echo

#Use SCP to transfer to file so you can verify successful backups -- Make sure to use identity/keys for SCP instead of a password:
scp -v -i /home/administrator/identity /home/administrator/scripts/sqldata.tgz administrator@this.host:/backup/sqldata_backup.tgz

perl /home/administrator/scripts/emailsql.pl

emailsql.pl (I did not write this one):

Instead of just sending the text of the log file, this script attaches the file and sends the message:


#!/usr/bin/perl -w
use MIME::Lite;

$msg = MIME::Lite->new(
From    => 'Backup Log',
To      => 'liveaverage@yourdomain.org',
Subject => 'MySQL Data Backup - Intranets',
Type    => 'text/plain',
Data    => "See the attached log for details on the most recent MySQL Database Dumps.");

$msg->attach(
Type       =>'text/plain',
Path       =>'/home/administrator/scripts/sql.backup.log',
Filename   =>'sql.backup.log',
Disposition        =>'attachment');

$msg->send;

Perl my way to happiness

JR — Fri, 25 Apr 2008 19:00:49 +0000

If only everything was as easy & straight-forward as account provisioning in Zimbra:

!/usr/bin/perl

# ZCS IMPORT SCRIPT
#       Provided by : ZCS Wiki
#       Modified by : J.R.
#       Last Change : 2008.04.09
#
# Lookup the valid COS (Class of Service) ID in the interface or like this
my $cosid = `su - zimbra -c 'zmprov gc apd |grep zimbraId:'`;
$cosid =~ s/zimbraId:\s*|\s*$//g;

while (<>) {
chomp;

# CHANGE ME: To the actual fields you use in your CSV file
my ($email, $password, $first, $last) = split(/\,/, $_, 4);

my ($uid, $domain) = split(/@/, $email, 2);

print qq{ca $uid\@$domain $password\n};
print qq{ma $uid\@$domain zimbraCOSid "$cosid"\n};
print qq{ma $uid\@$domain givenName "$first"\n};
print qq{ma $uid\@$domain sn "$last"\n};
print qq{ma $uid\@$domain cn "$uid"\n};
print qq{ma $uid\@$domain displayName "$first $last"\n};

#Set the user's reply or canonical address
print qq{ma $uid\@$domain zimbraMailCanonicalAddress $uid\@cityof*****.org\n};

#Add e-mail account alias for multiple domains
#Verify domain is correctly working for provisioning aliases

print qq{aaa $uid\@$domain $uid\@cityof*****.com\n};

#Add all users to a general distribution list and terminate
#Add multiple distro-lists if desired
print qq{adlm dept.all\@cityof*****.org $uid\@cityof*****.org\n};
print qq{\n};

}