Swim Team v1.42-beta-6 available

This morning I have posted a new build, v1.42 beta-6 of the Swim Team plugin.  This beta release makes a major change to how the plugin includes the various files the plugin uses.  Historically, Swim Team modified the PHP include_path as part of the initialization to include the paths where included files are expected.  In newer versions of PHP this seems to cause some problems for some users.

I have changed the plugin to eliminate the need to modify the PHP include_path however the process of doing so required touching almost 100 files within the plugin.  I have been through all of the menus and tabs and believe I found accounted for every file which included but there is always a chance I have missed something.

Please report any problems and I will do my best to fix it quickly.

wp-SwimTeam Beta (353 downloads)

Google Forms v0.66 available

This evening I released v0.66 of Google Forms.  This version fixes two major issues:

  1. The conflict with WordPress SEO has been resolved.  The implemented solution will likely be replaced with something which runs less frequently.  However, the solution implemented in v0.66 should make sites running both Google Forms and WordPress SEO a little happier.
  2. Due to a bug, all form submissions have been logged regardless of the plugin setting.

You can find the update on the WordPress plugin repository or on your Dashboard.

Google Forms v0.64-beta-3 available

This evening I uploaded v0.64-beta-3 of Google Forms.  This version adds a check for the WordPress HTTP API cURL transport and issue a notification if it isn’t present.

There was a substantial change to the WordPress HTTP API between 3.6.1 and 3.7.  I don’t know the full details of the change but what I found in pretty extensive testing is the streams and fsockopen transports work in 3.6.1 with the Google Forms plugin but from 3.7 and later, they do not.

However, the cURL transport does work and as long as it is available, the plugin will work as it always has.  The problem is I cannot find any way to work with sites that don’t have the cURL transport available so I have added this check and issue a notice when it isn’t present.

GForm_SS_70

 

Please download and test out this beta version and report any issues found.

Google Forms Beta (500 downloads)

Google Forms, Checkboxes and PHP 5.4 or 5.5

WordPress Google Form v0.63 introduced a change  which allowed the plugin to work on newer versions of PHP, notably 5.4 and 5.5.  Unfortunately that change has broken support for checkboxes.

For those who are new to the plugin or never needed to know how it worked, what the plugin does is retrieve the HTML for the form from Google and render it within the context of WordPress.  When the form is submitted, it is actually submitted within the context of WordPress.   The data is collected by the plugin and then submitted to Google.  The retrieval from and submission to Google is done with the WordPress HTTP API.  In particular, the wp_remote_get() and wp_remote_post() functions are used to retrieve and submit the form.

To complicate the problem further, Google uses Python as the backend for their form processor where as WordPress uses PHP.  For the most part, the fact that they are based on different scripting languages isn’t a big deal.  Until you get to checkboxes.  Checkboxes in Python are handled differently than they are in PHP.

I had solved the compatibility problem a couple years ago (see this thread on the wp-hackers mailing list) using a small jQuery script which fixed the form variables on the WordPress side and manual construction of the body parameter for wp_remote_post() when submitting the data to Google.  This solution worked fine until I received a bug report that nothing was being submitted on a site which was running PHP 5.4.x.

Fortunately the user who encountered the problem provided me with a patch that I was able to fold in which changed the way the body parameter was constructed (array instead of a string) which worked with PHP 5.4.x and also worked with older versions.  However, I didn’t test it thoroughly as I have had several reports that checkbox content was not being submitted correctly.  Uh-oh.  I was able to verify the problem fairly quickly and was able to push out a version which essentially reverts how the body parameter is constructed (string instead of an array).  The problem is, this build doesn’t work with newer versions of PHP.

I have PHP 5.5 running on an Ubuntu virtual machines for testing and so far, I have not found a solution which (a) works with PHP 5.5. and (b) submits checkboxes correctly.

Stay tuned.

Chasing down Email-Users out of memory issues

For the past couple of days I have been looking at two sites which were experiencing an issue with Email Users where any of the pages on the Dashboard which presents a list of recipients was incomplete.  Looking closer at the pages, PHP was crashing which resulted in an incomplete page.  One of the sites had another clue, it reported a request for memory which could not be granted.

Both users provided me access where I could upload a debug version of Email-Users.  While not an ideal debug environment, I am grateful for the trust and access both users provided as I would have not been able to chase this bug down in my own development environment.

I was able to narrow the problem down to a call to get_users() which is a standard WordPress API function.  Because Email Users has been around a while, it still contains some code which was necessary in older versions of WordPress and the arguments passed to get_users() included the ‘all_with_meta’ parameter which was the only way to retrieve the first and last name of a user prior to the magic methods which were introduced for get_users() in WordPress 3.x.  The magic methods remove the need for the ‘all_with_meta’ parameter however the plugin was never updated because it wasn’t broken.

In the process of chasing down this memory problem I added some code to partition the get_users() query into blocks of 500 users and I could watch the memory usage increase with each query.  This would continue until memory was exhausted at which time PHP would terminate ungracefully and the partial page would be rendered.

An email to the wp-hackers mailing list helped me understand how much data was being cached by calling get_users() with the ‘all_with_meta’ parameter and I realized I needed to find a different solution as what I was doing wouldn’t scale.

I had encountered the get_users() magic methods previously and I realized that I no longer needed to call get_users() with the ‘all_with_meta’ parameter so I removed it and did some testing and sure enough, I was able to successfully run on both sites without any issues.  Memory usage on the site with 13K users topped out at 47M, well under the 256M maximum defined by WordPress.

In the current beta version (4.6.3-beta-8) there is still some debug code in place to monitor memory usage.  If you look at the page source you will find something like this:

<pre id="line1"><!-- email-users.php::1091  Query #1  Memory Usage:  34.5M -->
<!-- email-users.php::1091  Query #2  Memory Usage:  34.75M -->
<!-- email-users.php::1091  Query #3  Memory Usage:  35.25M -->
<!-- email-users.php::1091  Query #4  Memory Usage:  35.75M -->
<!-- email-users.php::1091  Query #5  Memory Usage:  36.5M -->
<!-- email-users.php::1091  Query #6  Memory Usage:  37M -->
<!-- email-users.php::1091  Query #7  Memory Usage:  37.5M -->
<!-- email-users.php::1091  Query #8  Memory Usage:  37.75M -->
<!-- email-users.php::1091  Query #9  Memory Usage:  38.5M -->
<!-- email-users.php::1091  Query #10  Memory Usage:  39M -->
<!-- email-users.php::1091  Query #11  Memory Usage:  39.5M -->
<!-- email-users.php::1091  Query #12  Memory Usage:  40M -->
<!-- email-users.php::1091  Query #13  Memory Usage:  40.25M -->
<!-- email-users.php::1091  Query #14  Memory Usage:  40.75M -->
<!-- email-users.php::1091  Query #15  Memory Usage:  41.25M -->
<!-- email-users.php::1091  Query #16  Memory Usage:  41.5M -->
<!-- email-users.php::1091  Query #17  Memory Usage:  42.5M -->
<!-- email-users.php::1091  Query #18  Memory Usage:  43M -->
<!-- email-users.php::1091  Query #19  Memory Usage:  43.5M -->
<!-- email-users.php::1091  Query #20  Memory Usage:  44M -->
<!-- email-users.php::1091  Query #21  Memory Usage:  44.25M -->
<!-- email-users.php::1091  Query #22  Memory Usage:  44.75M -->
<!-- email-users.php::1091  Query #23  Memory Usage:  45.25M -->
<!-- email-users.php::1091  Query #24  Memory Usage:  45.75M -->
<!-- email-users.php::1091  Query #25  Memory Usage:  46M -->
<!-- email-users.php::1091  Query #26  Memory Usage:  46.5M -->
<!-- email-users.php::1091  Query #27  Memory Usage:  47M --></pre>

I need to do some additional testing but based on these two sites now working, I am bullish on this solution to this unusual problem.

Where did all of this wp-SwimTeam stuff come from?

You may have noticed an influx of content (posts, pages, etc.) related to wp-SwimTeam and wonder where it came from.  For the past four years I have been developing, enhancing, supporting, etc. a WordPress plugin called wp-SwimTeam.  This plugin can be used to manage the registration, volunteers, participation, and other aspects of running a youth swim team.  Several groups have used it for Masters Swimming as well but it is targeted at youth swim teams.

I first started wp-SwimTeam to support our local neighborhood team when my wife was the swim team chair.  Over the years it grew in features to the point where it was fairly comprehensive.  My children no longer swim and my involvement in swim team has largely ended but I still maintain my plugin because (a) people use it and (b) I still feel it serves a need and it doesn’t take much of my time to continue supporting it.

I have always had a separate web site for wp-SwimTeam but over time that came to make less and less sense. In fact, I had some people contact me on this site and some on the other site so I had questions and solutions on both sites.  I decided that I would migrate everything here So I exported all of the content and then imported it.  There were a few hiccups, mostly around downloads but I think everything else came over ok.

The old URL redirects to the wp-SwimTeam page on  this site.  From time to time you may see updates for wp-SwimTeam or phpHtmlLib in addition to the normal flow of information on Email Users and WordPress Google Form.

Email Users v4.6.0-beta-3 now available

If you’ve been following the Email Users Support Forum, you are aware that for the past couple of days I have been chasing a bug which affects some users and places all of their intended recipient email addresses in the CC header instead of the BCC header.

This has been a challenging bug to find as it doesn’t happen on any of the systems I have access to.  Fortunately I had a few users affected by this problem willing to help me and I appreciate their assistance.  I am not sure how I would resolved this problem without their assistance.  Again, thanks.

So what was the problem?  The WordPress wp_mail() API allows you to pass extra header information as part of the mail content.  Email Users makes use of this by constructing a BCC header which contains all of the email addresses for the intended recipients.  The wp_mail() function supports headers as a single string or an array of headers, each header as an array element.

Up until this evening Email Users had constructed all email headers as a string.  With a large number of email addresses this string could end up being very long.  However, I don’t believe the length of the string had anything to do with the problem.

When wp_mail() receives headers as a string, it tries to intelligently decompose them.  In some cases it appears that the string wasn’t splitting correctly which means the BCC field wasn’t ever seen.  The “Bcc:” text became part of an email address which was subsequently discarded because it resulted in a bad address.

Email Users v4.6.0 has a rewrite of the mailusers_send_mail() function plus some new debug functionality which helps resolve these sorts of issues.  Additionally, the new debug is under user control so can be turned on and off as needed.

Email Users Beta (243 downloads)

EU_SS_29

For testing this new version, I recommend turning Debug on until you’re comfortable with the the structure of the email addresses.

Questions or comments -don’t hesitate to ask.

Debugging the Email Users CC issue

For the past few days I have been chasing a reported issue where Email Users is putting recipient addresses into the CC header instead of the BCC header.  This has been a challenge to chase down.  To assist in this I have add a bunch of  new debug and checking to Email Users.

Email  Users will now check and report if wp_mail() has been overloaded by a plugin or theme.  Since wp_mail() is a pluggable function, this is the one area which is really out of the control of Email Users.  Simply knowing if wp_mail() has been overloaded is very helpful.

Additionally, I have finally found a way to initiate a complete email which is aborted just prior to the email actually being sent.  The content of several internal structures is displayed on the Dashboard so the email headers can easily be examined.

This build is in DEBUG mode which means it will never actually send an email unless wp_mail() has been overloaded in which case all bets are off.  Please visit the Settings page first to determine if wp_mail() has been overloaded.  If it has, do not send any email but let me know which plugin or theme has overloaded wp_mail().

Email Users Beta (243 downloads)

A Really Odd Email Users Problem

I was recently contacted about an Email Users issue on a site that is using bbPress and S2Member.  The person had come to the conclusion that there was some sort of conflict between the three plugins which resulted in the Email Users -> Users Settings screen being empty and asked me to look into it.

EU_SS_20

Clearly something is amiss as in order to access the WordPress Dashboard you need to login and in order to login, there needs to be at least one user!  This isn’t the sort of situation I can easily replicate.  Fortunately the person with this problem provided me access to their server so we could do some debugging.

The functionality in Email Users is based on the WP_List_Table class and an example plugin which you can find in the WordPress Plugin Repository.  When I first implemented it I was retrieving the list of users with a call to get_users() however I never got the sorting working correctly.  After playing around with it for a while, I gave up and wrote my own query.  The query I wrote yielded the results and sorting I wanted so I didn’t think much about it.  Until today.

This the query that WordPress generates and sends to MySQL:


SELECT ID, display_name, user_email, user_login, m1.meta_value first_name, m2.meta_value last_name, m3.meta_value massemail, m4.meta_value notifications
FROM wp_users u
LEFT JOIN wp_usermeta m1 ON (m1.user_id = u.ID AND m1.meta_key = 'first_name')
LEFT JOIN wp_usermeta m2 ON (m2.user_id = u.ID AND m2.meta_key = 'last_name')
LEFT JOIN wp_usermeta m3 ON (m3.user_id = u.ID AND m3.meta_key = 'email_users_accept_mass_emails')
LEFT JOIN wp_usermeta m4 ON (m4.user_id = u.ID AND m4.meta_key = 'email_users_accept_notifications')
ORDER BY last_name ASC LIMIT 0,100

What this query does is select all of the WordPress User Ids, first names, and last names, which have the proper Email Users user meta keys. The data is returned and stored in a format which can be rendered in the List Table.

Because the table was empty, I wanted to see what the results of the query were as that would tell me where the issue might be.  Because I had access to the MySQL database via phpMyAdmin, I was able to run the query directly against MySQL.  When I did so, this is what I found:

EU_SS_21

This certainly was unexpected but it explained why the List Table was empty.  I have never run into this sort of problem, in fact, I had never heard of either the SQL_MAX_JOIN_SIZE or the SQL_BIG_SELECTS MySQL system variables.

I was able to alter SQL_BIG_SELECTS using the following SQL:


SET SQL_BIG_SELECTS=1

When I did this and re-ran the query, phpMyAdmin returned the results I would have expected to see.  This was good news – it appears this users problem is due to a MySQL setting.  The bad news is sometimes it is hard to change these settings and it isn’t something I thought I could add to the query itself.

As a test, I modified the source for one of Email Users files so it was calling the original method I had coded  based on get_users() to see if it would work and it did.  This is very good news as I suspect fixing the sorting issue (or living with it) will be easier than changing the MySQL system variable settings.

My next step is to see if I can fix the sorting problem and roll out an update.

Email-Users Debugging – why don’t my users receive email?

Periodically users post questions in the Email-Users support forum wondering why their users don’t receive email messages when they expect them to.  There are many reasons why users don’t receive email, some are within the control of Email-Users but many are not.

Email-Users User Settings

Email-Users is based on two user settings (which appear on the user’s profile page):

  1. Receive post or page notification emails
  2. Receive emails sent to multiple recipients

EU_SS_14Email-Users has a plugin level setting which allows end users to modify their own settings which is enabled by default.  When it is disabled, only users with the proper permissions can modify user settings related to Email-Users.  No matter how the user has their settings configured, a user can always receive an email when they are the only recipient.

EU_SS_15In Email-Users v4.5.2 a new Dashboard Widget has been added which reports the number of users who have their settings configured to receive each type of email.  This information also appears on the Plugin Settings Page and should either of the types of email not have any users set to receive email, a warning message is displayed at the top of the screen.

EU_SS_17Changing the Email-Users settings for a large number of users can be accomplished from the User Setting menu.  By selecting the desired Users, Notifications and Mass Email can be enabled or disabled for a large number of users at one time.  For a site that has a large number of users, the number of users which appear on each page can be changed on the Settings Page.

Test Email

Before sending out notifications or messages, it is important to validate the underlying WordPress email system is working.  On the Plugin Settings page there is the ability to send a Test Message.

EU_SS_18

Email-Users utilizes WordPress’ wp_mail() function to send email.  It does not interact with the OS or PHP mail functions directly.  By using wp_mail(), which can be overridden by other WordPress plugins, Email-Users can work with different server configurations and email infrastructure.

If you don’t receive the test message then you may have a problem with your underlying WordPress email system.  Do you receive email for user registration and comments?  If not, there is something wrong which is out of the scope of Email-Users.

On some servers plugins such as WP-Mail-SMTP (this is just an example, there are others) can be used in situations where email is being sent but isn’t being received.

There is no point in trying to send Notifications or Mass Emails if the Test Message isn’t working.

BCC Limits

Some servers limit the number of email addresses which can appear in the BCC header of an email message.  Email-Users has a setting to control the number of addresses which appear in the BCC field.  If Email-Users is working for a small number of users but isn’t working for a larger number (e.g. all of your users) , you may be dealing with a BCC limit.

If you are able to send email to 3-5 users by selecting them from the list of users, I would next try sending a message to about 10 users and see if that works.

The sending logic is only different when 1 user is selected. Anytime more than one user is selected (2, 10, 37, 67, 1000, etc.), the same logic path through the code is used.

The only thing that is different is how many emails are sent and that is based on the BCC limit setting. If there is no limit, which is the default, it sends one email with all recipients on the BCC line. For any other setting, it will send some number of emails, depending on the BCC limit. If the BCC limit was set to 30 and you had 67 recipients, it would send 3 emails: The first 30, the second 30, and the final 7.

Advanced Debugging

In some situations you may want to examine what Email-Users is doing in gory detail.  For development purposes Email-Users has a DEBUG mode.  When in debug mode Email-Users will log quite a bit of information to the PHP error console using error_log() including the actual email headers it generates.  It can be very useful to see the actual mail headers to ensure the expected recipients are actually included.

Because gaining access to the contents of the error_log() differs from web host to web host, I recommend using a WordPress debugging capability to capture it in the site’s /wp-content folder.  By setting WP_DEBUG_LOG to true in your wp-config.php file, you can easily access the resulting debug.log file.  On my development machine I use the following set up in my wp-config.php file:

define('WP_DEBUG', true); // or false
if (WP_DEBUG) {
 define('WP_DEBUG_LOG', true);
 ini_set('display_errors', 'on');
 error_reporting(E_STRICT | E_ALL);
 define('SAVEQUERIES', true);
 define('SCRIPT_DEBUG', true);
 error_log('Debug On!') ;
}

You would not want this on a production server but to temporarily chase down a problem, it works well.  Unfortunately some plugins issue a lot of notices or warnings so you can run into the “Headers already sent” problem so you may want to tweak the above settings to meet your needs.

Once your WordPress debugging settings are tuned to your liking, you need to enable Debug for Email-Users.  Doing this requires manually editing one of the plugin files.

On or about line 46 of the email-users.php file you will find a line of code which reads:

define( ‘MAILUSERS_DEBUG’, false);

You need to change the code such that it reads:

define( ‘MAILUSERS_DEBUG’, true);

After you have made this change Email-Users will output information to the error_log.

In the image below you can see a series of email headers each with 10 email addresses in the BCC field (BCC limit is set to 10).

EU_SS_19

If the information in the error_log is what is expected then there is little, if anything which can be done from Email-Users.  At this point you will need to figure out where the email is routing to and where it isn’t which is beyond the scope of the Email-Users plugin.

Anecdote

I was using Email-Users on our local Swim Team’s web site and heard from a number of parents that they weren’t receiving emails.  I noticed that all of the reports were coming from families with RoadRunner email addresses.  It turned out that Time Warner had blacklisted our Swim Team’s domain.  By filling out a form with TWC and verifying some information, we were able to get white listed but for several weeks I wasn’t sure how to resolve it.