« December 2010 | Main | February 2011 »

January 31, 2011

Cleaning up the data

Now that the housing application is "done" I am going to catch up on data stuffs. Things that need to happen:

- Switch back to DataSet/DataFile approach
- Update Panel template and Views to pull info from DataFile
- Clean up data from REUs and Frontiers students
- Load data from Bob
- Load bibliography entries from Bob
- Break for lunch...

Posted by kkwaiser at 04:37 PM | Comments (0)

Reset AutoIncrement counter in MySQL

See this comment:

ALTER TABLE [tablename] AUTO_INCREMENT = [number]

Posted by kkwaiser at 03:29 PM | Comments (0)

Reordering MySQL columns

See this post.

Posted by kkwaiser at 02:16 PM | Comments (0)

January 28, 2011

Script to form CSV for database insertion

My technique will almost assuredly become more refined as time passes but, for now, here is a simple script that takes in a CSV file and outputs an SQL statement.

#! /usr/bin/python


import csv, sys

inDir ='/home/data/Desktop/'
inFile = 'test_datafile.csv'
db = "umbs"
table = "temptable"

#Open CSV into a list
in1 = list(csv.reader( open('{0}{1}'.format(inDir, inFile), 'r') , delimiter=','))
out = open('{0}test_output.csv'.format(inDir), 'w') # output Document
#See http://docs.python.org/tutorial/inputoutput.html for discussion on .format

#Write INSERT, db and table, column names and begin VALUES
out.write("INSERT INTO {0}.{1} ".format(db, table) +"\n")
out.write("(" + str(in1[0]).strip('[]') + ")\n")#First row must match table fields

#Write the data (i.e., rows 2:End
for row in in1[1:]:
if row != in1[-1]:
out.write("(" + str(row).strip('[]') + "),\n")
out.write("(" + str(row).strip('[]') + ");\n")

#Close the output file

except IOError:

print 'Can\'t open file for reading.'


Posted by kkwaiser at 03:39 PM | Comments (0)

January 26, 2011

Rebuilding Drupal caches

This is mostly a bookmark right now but at some point I want to figure out how to programmatically rebuild caches in Drupal. Caches expire and don't seem to rebuild until someone goes to view a page, for a page such as our research site map this means a 15 second load time.

The thread discusses how to accomplish cache rebuilds via a custom module.

Here's a informative section on Cron and Caching.

Posted by kkwaiser at 09:07 AM | Comments (0)

January 20, 2011

Known "bugs" and bugs in the housing application

Here goes:

1) Clicking "Add another item" two times creates undefined solution
- This basically means the applicant will have to answer the captcha question twice instead of once.
- Severity: minor nuisance
- Solution: The module developers are aware of the problem, hopefully a fix with be forthcoming.

2) Applicants must have their browser's JavaScript enabled for the form to work.
- The latest Google Analytics report indicates 91-93% of visitors to our website have JavaScript enabled.
- Severity: potential headache
- Solution: Hire a very expensive programmer, go to PDF application, use last year's application (which only barely functions without javaScript)

3) Spam filters might eat some of the auto-emails?
- Problem occurred with REU applicants but may not happen here.
- Severity: Won't affect our processing or reception of housing applications. If researchers begin creating accounts and not receiving autoemails then this will become a bigger problem.
- Solution: I've adjusted the text of autoemails in an attempt to not trigger spam filters. Further action may be possible but will require more research.

4) If you submit, backup, submit, backup enough times the hidden fields (e.g. guest fields) will appear even if they should be hidden.
- Cannot reproduce this on a regular basis
- Severity: Minor
- Solution: Don't backup and submit several housing applications!

Posted by kkwaiser at 01:23 PM | Comments (0)

January 19, 2011

Code Snippet for pulling Node Reference and Multigroup information in Drupal 6

Saving are the snippets.

Here's the Node reference snippet. It loads a referenced node and prints out some of the fields.

//Print Guest Information

echo "------------------- Guest Information ------------------- \n";
$guest_ref = $node->field_housing_guest_ref;

foreach( (array)$guest_ref as $item){
$guest = node_load($item[nid]);
$guest_arrival = explode('T', $guest->field_housing_guest_dates[0][value]);
$guest_departure = explode('T', $guest->field_housing_guest_dates[0][value2]);

echo "Guest name: " . $guest->field_housing_guest_fname[0][value] . " " . $guest->field_housing_guest_lname[0][value] . "\n";
echo "Arrival: " . $guest_arrival[0] . "\n";
echo "Departure: " . $guest_departure[0] . "\n\n";

Here is the multigroup example:

//Print Guest Information
echo "------------------- Guest Information ------------------- \n";

$guests_fname = $node->field_housing_guest_fname;
$guests_lname = $node->field_housing_guest_lname;
$guests_age = $node->field_housing_guest_age;
$guests_dates = $node->field_housing_guest_dates;

$i = 0;
foreach($guests_fname as $item){
$guest_arrival = explode('T', $guests_dates[$i][value]);
$guest_departure = explode('T', $guests_dates[$i][value2]);

echo "Guest: " . $item[value] . " " . $guests_lname[$i][value] . "\n";
echo "Arrival: " . $guest_arrival[0] . "\n";
echo "Departure: " . $guest_departure[0] . "\n\n";


echo "----------------------------------------------------------------- \n\n";

Posted by kkwaiser at 10:43 AM | Comments (0)

January 18, 2011

Downgrading from PHP 5.3 to 5.2 all for Drupal

Contributed modules in Drupal 6 don't like PHP 5.3. For example, I'm pretty sure CCK's export and multigroup views integration hate it.

Notes on backing down the step - or jumping off the cliff, we'll see how it turns out. This is on Ubuntu 10.04 btw.

Newbie friendly tutorial

Drupal friendly tutorial

Instructions for Ubuntu 10.10

Details I didn't quite understand

$ wget http://randyfay.com/sites/default/files/karmic_0.list
$ wget http://randyfay.com/sites/default/files/php_1.
$ sudo mv karmic_0.list /etc/apt/sources.list.d/karmic.list
$ sudo mv php_1. /etc/apt/preferences.d/php

$ dpkg --get-selections | grep php
$ sudo apt-get remove libapache2-mod-php5 php5 php5-cli php5-common php5-gd php5-mcrypt php5-mysql

$ sudo apt-get autoremove
$ sudo apt-get update
$ sudo apt-get install php5 libapache2-mod-php5 php-db php5-cli php5-common php5-xsl php5-gd php-pear php5-mysql php5-curl php5-memcache

Add three lines to /etc/apt/preferences.d/php

Package: php5-mcrypt
Pin: release a=karmic
Pin-Priority: 991

$ sudo apt-get mcrypt

$ php -v

Modify /etc/php5/apache2/php.ini

max_execution_time = 200
memory_limit = 128M
log_errors = On
error_log = /var/log/apache2/php_errors.log

And...BINGO, WAMBO, KAPOW, I can now export content types in Drupal again.

Posted by kkwaiser at 12:51 PM | Comments (0)

Logging php errors

Note schmotes...

Configure php.ini

$ sudo nano /etc/php5/apache2/php.ini

log_errors = On
error_log = /var/log/apache2/php_errors.log

$ sudo /etc/init.d/apache2 restart

View the log:
$ less /var/log/apache2/php_errors.log

Posted by kkwaiser at 11:26 AM | Comments (0)

January 17, 2011

Further modification of the housing form

Notes, notes, notes

1) Patched core so that it would quit messing with my hidden, required radio buttons. It was screwing with the post-submission validation step. More here.

2) Made Guests its own content type (rather than use CCK3 multigroup which is Views-impaired). Using Node Relationship and Modal Frame with a node reference field to allow guests to be added.

3) Made the Guest node reference field unrequired due to validation errors. Report here.

4) Auto-emails are set up to fire to registrant and UMBS staff, need to tweak the php in these emails.

5) Upon submission, need to redirect to generic confirmation page.

Posted by kkwaiser at 09:57 AM | Comments (0)

Avoiding spam filters

Blogpost hijack...begin!

Installed the SMTP module and am optimistic my problems are now ALL solved. To wit:

Original Header:

Received-SPF: pass (google.com: best guess record for domain of www@[server1].umich.edu designates [ip1] as permitted sender) client-ip=[ip1];

New Header

Received-SPF: pass (google.com: best guess record for domain of [site_email]@umich.edu designates [ip1] as permitted sender) client-ip=[ip1];

Original Header:

Received: (from www@localhost)
by [server1].umich.edu ( id p0OIMVP3003713;
Mon, 24 Jan 2011 13:22:31 -0500

New Header

Received: FROM umbs.lsa.umich.edu ([server1].umich.edu [ip1])
By [server2].umich.edu ID 4D3EE9D9.6ED0B.21456 ;
Authuser kkwaiser;
25 Jan 2011 10:18:49 EST

Blogpost hijack...end!

Email I sent to www-sig:

A not insignificant portion of auto-emails coming from my website (Drupal install hosted on an ITS Virtual Server) are being flagged as spam. The emails are generally user-confirmation or password reset emails. Is there anything I can do to increase the likelihood that these emails go through to the recipient without being caught in spam filters?

See below the fold for the responses I received. I will focus on changes I've made next.

Change subject line of auto-emails

Account confirmation:

- Original: Account details for [username] at University of Michigan Biological Station
- New: Registration confirmation at the University of Michigan Biological Station

Password recovery:

- Original: Replacement login information for [username] at University of Michigan Biological Station
- New: Replacement login information at the University of Michigan Biological Station

Basically, I'm hoping that removing the username from the mix will do the trick. I've also removed username references from the body of these emails.

Check to see if the email servers I'm using are flagged as spam sources

1) Look at email header to determine IP address of email servers. This page is a good source of information.

2) Plug IP Addresses into a web application that checks against listings of spam sources.

3) ITS provides a similar service for spam blockers on the UM campus.

None of the servers I am using were flagged so I'm going to focus on altering the structure of the auto-emails at this point.


I'm not optimistic that the above changes are going to have much affect and it is too important that our emails get to where they need to be.

I've run DNS reports (after reading this thread) at this site and this site for umbs.lsa.umich.edu and sent them on to ITS Webmasters for feedback. Hopefully, this will help.

This thread discusses DNS report results that may indicated spam filtering problems.

Change "From" email address:
Original: umbs-research-gateway@umich.edu
New: umbs-research-gateway@umbs.lsa.umich.edu

Response 1:

There are several things you can do. First is to avoid the use of words, or combinations of words, which can trigger the Bayesian filters. Lists of common spam words are available via Google. Another thing is to be sure that your outgoing emails are properly formatted. Formatting errors with RTF/HTML/etc. can flag messages. Header formatting is also an issue. Another is of course attachments.

One thing you can do is check the headers of the messages that get flagged. Many filters attach a header (e.g. X-SPAM) with the "hit count" (instances of triggering the filters) for each test. That might tip you off as to what in particular is causing the messages to be flagged.

Hope that helps. Please let me know if there is anything else I can do.

Response 2:

Do what Jaime already said first (and note that if the message recipient uses the ITS IMAP service, spam factors are off by default, but the recipient can have them turned on for all future messages by contacting imap-postmasters@umich.edu).

In addition to what [the previous emailer] already said, you can check the IP address from which your message is being sent (N.B.: NOT the IP address of your website, nor your domain name, but the IP from which the email is actually originating) to see if it has a bad reputation as an IP address from which spam originates. Check both the realtime blackhole lists that U-M uses ( http://spambusters.mail.umich.edu/troubleshoot/blockstatus/ ) as well as others worldwide (google for both "rbl" and "ip reputation" to find quite a few to check). If you discover you're on a blacklist or the IP in question has a bad reputation, contact postmaster@umich.edu and work with them to resolve the issue.

I advise against pursuing things such as SPF or DKIM for your own domains outside of umich.edu unless you have hard evidence that using one of these will in fact result in significantly better message tagging and acceptance by major ISPs. U-M uses neither, by the way, due to cost/benefit reasons, and I have not heard of any problems resulting from the lack up to this point.

I hope this is useful.

*Emphasis added by me.

Posted by kkwaiser at 09:00 AM | Comments (0)

January 14, 2011

Major impediments re: Housing Application


At least three of them?

CCK3 (multigroup)
- great for getting data in but good luck using it with views. Increasingly think this is a dead-end for D7


CCK 6.x

If a user "Adds Another" and then "Removes this item" on a required field, Drupal's form validation throws any errror - even if the user has another field completed for that field. The error has been on the CCK issue topic for 4 years and is not yet resolved.

Conditional Fields 6.x-2.x

- Validation of form fails if a hidden, required field is not complete. Appears the problem is only with the 6.x-2.x branch

Conditional Fields 6.x-1.x

The above problem does not seem to appear in this branch but it is impossible to have multiple fields control the same field per this issue.

Posted by kkwaiser at 12:29 PM | Comments (0)

January 11, 2011

Setup Google Analytics for Drupal6

Here are the steps (as I remember them). You don't need to add the code to every page:

1) Install and enable Google Analytics
2) Check permissions
3) Configure GA (i.e., specify roles to track)
4) Register site with GA
- Create New Website Profile (enter URL)
- Copy Web Property ID to GA config in Drupal
- Verify Site Ownership? - This part is a bit fuzzy but you may need to place an html file in the root web directory to prove you are the site admin.

5) Configure reports and wait. It will take up to 1-2 days for data to kick-in.

Posted by kkwaiser at 10:10 AM | Comments (0)

January 10, 2011

Devel conflict with "Add Another" button on Drupal forms

A note that the Devel Module conflict with the Add Another button that shows up on content types with an unlimited number of entries. See more here.

Posted by kkwaiser at 01:59 PM | Comments (0)

Drupal update notes

Nothing to see here except a few reminders for a round of updates I need to run:

WebForm 3.x needs to be updated. Currently disabled due to highly critical vulnerability

WYSIWYG - after update will need to either 1) patch module to remove unwanted breaks in blank text boxes OR 2) update CKeditor which may have fixed the bug

I've run a full drush update on the test site

Posted by kkwaiser at 09:31 AM | Comments (0)

January 06, 2011

How to for Drupals Devel Module

Post 201!!

I've been wanting to write this one for a while. This will be my second install and use of the Devel Module. Here are my steps:

1. Download and enable Devel
- No sub-modules needed at this point
- run update.php

2. Check/adjust permissions (admin/user/permissions)
- Give admin everything

3. Check Devel settings (admin/settings/devel)
- defaults for now

4. Enable Devel-created Blocks (admin/build/block)
- Development and Execute PHP are the blocks I've enables. Be sure to make them visible to the Administrator role and NOBODY ELSE.

- For some unjustifiable reason these blocks only appear on the block admin page (Marinelli theme) but not on any other pages (Garland theme).


Nevermind, apparently when you have multiple themes on a site you need to enable the block on each theme. This post tipped me off.

5. Further notes
- After enabling Devel, each node will have a new Devel tab where you can see the components of the node (e.g, arrays and shit), tokens and other stuffs.

Posted by kkwaiser at 03:33 PM | Comments (0)