Mar 12th, 2009 | Perl, Programming, Technology | No Comments
Recently I had to do an import of mailboxes hosted in the maildirs into RT3. Looking around on the web didn’t found ready solutions. Looking some more and getting small portions of code here and there ended up writing my own script.
One thing though, I had to stop using rt-mailgate for the purpose of import since it was overloading Apache and MySQL. Through I still use rt-mailgate for normal mail aliases to pass incoming mail to RT.
The script should be started from inside of the root of the maildir which you want to import. Prior to this, few parameters needed to be adjusted like the RT queue, tickets status and log file (right at the top of the script).
Few notes:
- assuming RT is hosted on the same machine the maildir is located (alternativly copy the maildir to the same server).
- script works with files and RT modules (which work directly with MySQL) and is much lighter and more flexible than the default rt-mailgate.
- I am not a programmer, so don’t blame me for bad coding, suggestions welcome.
- All the stuff is in perl, tested to be working for me on Fedora 10 with RT3 version 3.8.2
- If you uncomment the DEBUG section of the code, it will do dry run, showing what will be imported and what not, without doing actual imports or writing to any logs.
- If, by some chance, you lost your log of previous import, or want to import only a portion of emails in the current maildir, you can use unix find instead of perl glob in the for statement. (For example, put the next statement as a condition for for loop to find messages that were changed within last day: split /\n/,`find new cur tmp .*/cur .*/new .*/tmp -mtime -1 -type f`)
Ok, here is the code:
#!/usr/bin/perl -w
use RT;
use Email::MIME;
use Data::Dumper;
# RT Queue to import email to
my $desired_queue = "my_rt_queue_name_here";
# Log file to write update (and read old data from)
my $log_file = "/tmp/rt3_import_log-$desired_queue.txt";
# Status of the tickets created
my $status = 'resolved';
# Get connected to RT
RT::LoadConfig();
RT::Init();
RT::ConnectToDatabase();
# Store already imported emails here
my $imported = {};
# If the log file already exists
if (-f $log_file) {
# Try to read it and get all email ids
# that were already imported
open LOG, "< $log_file";
while (my $line = ) {
if (my $id = get_msgid($line)) {
$imported->{$id} = 1;
}
}
close LOG;
}
open LOG, ">>$log_file";
my $total = 0;
# Find all files in current dir (recursivly
for my $file (glob "cur/* tmp/* new/* .*/cur/* .*/tmp/* .*/new/*") {
$total++;
print "$file: ";
# Try to get message ID
my $msgid = get_msgid($file);
if ($msgid) {
# Skip if already imported
if (defined($imported->{$msgid}) && $imported->{$msgid} == 1) {
print "skipping\n";
next;
}
} else {
# Skip of no message ID
print "no message id\n";
next;
}
# DEBUG!!!
# print "fetching\n";
# next;
# !DEBUG
# Try to create a ticket
my ($id,$error) = create_ticket($desired_queue,$file,$status);
if ($id) {
# Log to STDOUT and log file on success
print " $id\n";
print LOG "$file: $id\n";
$imported->{$msgid} = 1;
} else {
# Log to STDOUT on failure
print "$error\n";
}
# Sleep for 30 secs each 50 msgs not to overload MySQL
# (Adjust if needed, this is for heavy production systems)
if ($total == 50) {
$total = 0;
sleep 30;
}
}
close LOG;
# Parse message ID from file name
sub get_msgid {
my $msg = shift;
if ($msg =~ /^.*\/([0-9A-Z]+)\.([0-9A-Z]+)\.[^\/]+?:.*$/) {
# Return only the first two portions of msg ID
# (works file with Exim/Dovecot IDs)
return "$1.$2";
}
return 0;
}
# Create a ticket in RT
sub create_ticket {
my ($queue,$filename,$ticket_status) = @_;
# Read the content of the msg file
open FH,"< $filename";
my $message = "";
my $subject = "";
while (my $line = ) {
$message .= "$line";
# Try to find out the subject
if ($line =~ /^Subject: (.*)\n$/ && $subject eq "") {
$subject = $1;
}
}
close FH;
# Create MIME entity (RT wants it like this)
my $entity = new Email::MIME($message);
# Parse it the way RT wants
my $parser = RT::EmailParser->new();
$parser->SmartParseMIMEEntityFromScalar(Message => $entity->as_string);
# Create the ticket
my $ticket = new RT::Ticket($RT::SystemUser);
my ($t_id,$transaction,$error_str) = $ticket->Create(
Queue => $queue,
Requestor => $entity->header('From'),
Subject => $subject,
MIMEObj => $parser->Entity,
Status => $ticket_status,
);
# Give back ID and message
return ($t_id,$error_str);
}
Jan 28th, 2007 | Blogging, Education, General, Java, Perl, Personal, Programming, Technology | 2 Comments
It’s been a while since I last posted here and I don’t feel like posting something huge here this time, so here is just a short life update:
- Wrote all college exams (4 of them):
- Artificial Intelligence – this is the only exam I feel like I failed. The subject is really boring, the teaches does not seems to be confident in the topic, exams were prepared by other professor who is very good in subject and probably did not consider that (as I believe) we were taught not in the best way. In addition the class is kinda boring. As I have concluded – it is like maths for me, but while in maths I know that I am totally stupid, here you think for the whole semester that the class goes fine and at the end you find yourself unbelievable stupid.
- Software Engineering – this one went fine. A class is very boring and implies a lot of bureaucracy related to IEEE standards, but I think I managed it quite good mostly because IEEE has a lot in common with ISO stuff which I know not that bad.
- Computer Architecture and Organization – went very easy because the amount of information given was very limited and a lot of it came out of previously passed Digital Systems.
- Object Oriented Programming (based on Java) – went pretty good as well due to some programming experience I have as well as familiarization with many programming languages. One more time proved to myself that Java is not what I like – Perl is the best.
- Preparing for the next semester – need to pay quite an amount of money for the college but already have some options. Thanks to all people who help me in this matter.
- Learning AJAX technology and have a post about it on configfun.com here.
- Preparing for the visit to immigration :( Hate this part of being in Cyprus, but have no other ways yet.
- Planning to go for skiing one day into Troodos where there is some snow already. Hopefully my skiing equipment will be delivered to Cyprus by a friend of mine who is now in Moscow.
Dec 27th, 2006 | Nagios, Perl, Programming, Software, Technology | 8 Comments
I am currently working on creating a perl module do deal with Nagios configuration. The existing Nagios-Object package is quite limited, outdated and not that fully implemented. I checked it up and decided to rewrite it using my way.
Currently I am using Class-Generate package to help me with creating classes for Nagios object definitions and I already managed to write a parser which takes a path of nagios main configuration file, finds all links to object definition files and parses all definitions. One’s parsed, the definitions are stored as objects with all accessorts/mutators available.
I am also implementing object linkage so that the relations of objects can be used easily. Currently most of the stuff is already working fine.
In addition I am planning to implement the write function so that it is possible to create problem text config files with object definitions. Also thinking on getting on status log parsing.
The idea behind is to create a web admin panel for Nagios and extend the functionality, plus maybe I will do a new web frontend for Nagios, since the one given is not that good (anymore). I already have some web stuff working (based on perl, Catalyst, CGI-FromBuilder and Template-Toolkit)
Dec 11th, 2006 | Blogging, Perl, Programming, Technology | 2 Comments
JRB Technology blog offers a script on how to backup a MySQL database. The described solution uses perl to call three commands:
@b1 = `mysqldump -h localhost -u mysql_username –password=mysql_password mysql_database > /path/to/put/mysql_database.sql`;@z1 = `zip -r -9 ~/path/to/mysql_database.zip ~/path/to/mysql_database.sql`;@d = `rm -f ~/path/to/mysql_database.sql`;
I have posted a comment directly to the origin blog with some (as I see it) improvements, but unfortunately the comment was deleted, so I will reproduce it here.
First of all I think that there is no need to create a temporarily SQL file with dump from the database and then zipping it. The zip command supports getting data from STDIN, so this way, instead of running 3 commands, only one is needed:
mysqldump -h hostname -u username –password=somepassword databasename | zip -9 mysql_database.zip
Then I offered to hide a password, since otherwise it is possible to see it by running the ps command under any user during script execution, and password will be seen in the arguments list. To hide a password, we can create a text file which is accessible only to us (chmod 600) and store a password string there, then modify the backup command accordingly:
mysqldump -h hostname -u username –password=`cat /path/to/securefile.txt` databasename | zip -9 mysql_database.zip
This way, even if someone will see that the password for accessing database is in the specified text file, it will be harder to get it, since file is readable only by owner user.
Finally, I would like comment a bit on Perl script itself, which author of the original script provides: first of all, I don’t see any point of using Perl here, since nothing, except for calling standard commands is done during execution. A plain bash script would do a job fine. Anyway, even if you prefer using Perl, I would strongly recommend using -w switch and use strict; statement (even in such simple script, it is just good to use this every time)
Nov 9th, 2006 | Education, Java, Linux, OS, Perl, Personal, Programming, Technology | No Comments
I am not a Java fan and I do prefer doing all my stuff in Perl (due to the perl being more flexible and comfortable to code in [my personal view] and probably due to the nature of the tasks I have), but due to the college class Object Oriented Programming which is totally taught in Java, I have to install Sun JDK on my laptop for some time.
It turned out that the task is not that easy (actually easy, but tricky), but I found a nice article which describes all steps needed to get Sun JDK configured on the system – works for me