MailDir to RT import

Recently I had to do an import of mailboxes hosted in the maildirs into RT3. Looking around on the web didn\’t found ready solutions. Looking some more and getting small portions of code here and there ended up writing my own script.

One thing though, I had to stop using rt-mailgate for the purpose of import since it was overloading Apache and MySQL. Through I still use rt-mailgate for normal mail aliases to pass incoming mail to RT.

The script should be started from inside of the root of the maildir which you want to import. Prior to this, few parameters needed to be adjusted like the RT queue, tickets status and log file (right at the top of the script).

Few notes:
– assuming RT is hosted on the same machine the maildir is located (alternativly copy the maildir to the same server).
– script works with files and RT modules (which work directly with MySQL) and is much lighter and more flexible than the default rt-mailgate.
– I am not a programmer, so don\’t blame me for bad coding, suggestions welcome.
– All the stuff is in perl, tested to be working for me on Fedora 10 with RT3 version 3.8.2
– If you uncomment the DEBUG section of the code, it will do dry run, showing what will be imported and what not, without doing actual imports or writing to any logs.
– If, by some chance, you lost your log of previous import, or want to import only a portion of emails in the current maildir, you can use unix find instead of perl glob in the for statement. (For example, put the next statement as a condition for for loop to find messages that were changed within last day: split /\\n/,`find new cur tmp .*/cur .*/new .*/tmp -mtime -1 -type f`)

Ok, here is the code:

#!/usr/bin/perl -w

use RT;
use Email::MIME;
use Data::Dumper;

# RT Queue to import email to
my $desired_queue = \"my_rt_queue_name_here\";

# Log file to write update (and read old data from)
my $log_file = \"/tmp/rt3_import_log-$desired_queue.txt\";

# Status of the tickets created
my $status = \'resolved\';

# Get connected to RT
RT::LoadConfig();
RT::Init();
RT::ConnectToDatabase();

# Store already imported emails here
my $imported = {};

# If the log file already exists
if (-f $log_file) {

	# Try to read it and get all email ids
	# that were already imported
	open LOG, \"< $log_file\";
	while (my $line = ) {
		if (my $id = get_msgid($line)) {
			$imported->{$id} = 1;
		}
	}
	close LOG;
}


open LOG, \">>$log_file\";
my $total = 0;

# Find all files in current dir (recursivly
for my $file (glob \"cur/* tmp/* new/* .*/cur/* .*/tmp/* .*/new/*\") {

	$total++;
	print \"$file: \";

	# Try to get message ID
	my $msgid = get_msgid($file);
	if ($msgid) {

		# Skip if already imported
		if (defined($imported->{$msgid}) && $imported->{$msgid} == 1) {
			print \"skipping\\n\";
			next;
		}
	} else {

		# Skip of no message ID
		print \"no message id\\n\";
		next;
	}

# DEBUG!!!
#	print \"fetching\\n\";
#	next;
# !DEBUG

	# Try to create a ticket
	my ($id,$error) = create_ticket($desired_queue,$file,$status);


	if ($id) {

		# Log to STDOUT and log file on success
		print \" $id\\n\";
		print LOG \"$file: $id\\n\";
		$imported->{$msgid} = 1;

	} else {

		# Log to STDOUT on failure
		print \"$error\\n\";
	}

	# Sleep for 30 secs each 50 msgs not to overload MySQL
	# (Adjust if needed, this is for heavy production systems)
	if ($total == 50) {
		$total = 0;
		sleep 30;
	}
}

close LOG;

# Parse message ID from file name
sub get_msgid {
	my $msg = shift;

	if ($msg =~ /^.*\\/([0-9A-Z]+)\\.([0-9A-Z]+)\\.[^\\/]+?:.*$/) {

		# Return only the first two portions of msg ID
		# (works file with Exim/Dovecot IDs)
		return \"$1.$2\";
	}

	return 0;
}

# Create a ticket in RT
sub create_ticket {
	my ($queue,$filename,$ticket_status) = @_;

	# Read the content of the msg file
	open FH,\"< $filename\";
	my $message = \"\";
	my $subject = \"\";
	while (my $line = ) {
		$message .= \"$line\";

		# Try to find out the subject 
		if ($line =~ /^Subject: (.*)\\n$/ && $subject eq \"\") {
			$subject = $1;
		}
	}
	close FH;

	# Create MIME entity (RT wants it like this)
	my $entity = new Email::MIME($message);

	# Parse it the way RT wants
	my $parser = RT::EmailParser->new();
	$parser->SmartParseMIMEEntityFromScalar(Message => $entity->as_string);

	# Create the ticket
	my $ticket = new RT::Ticket($RT::SystemUser);
	my ($t_id,$transaction,$error_str) = $ticket->Create(
			Queue => $queue,
			Requestor => $entity->header(\'From\'),
			Subject => $subject,
			MIMEObj => $parser->Entity,
			Status => $ticket_status,
	);

	# Give back ID and message
	return ($t_id,$error_str);
}