A small enough to not really be noticed at first, so it goes on long enough that the server gets blakclisted everywhere. 

The thing with this, is that it wasn't a standard SPAM script that was getting hit, and generating spam, or a hacked email account.
This was basically EVERY domain on the server (Almost all owned by one reseller on the server) all sending one or two pieces of spam here and there from info@<customerdomain>

The mail logs would just show generic information like

Mail Control Data:

mailnull 47 12
<info@(customerdomain)>
1483750320 0
-helo_name (customerdomain)
-host_address 127.0.0.1.57428
-interface_address 127.0.0.1.25
-received_protocol esmtp
-aclc _authenticated_local_user 6
nobody
-body_linecount 16
-max_received_linelength 390
-host_lookup_failed
XX
1
(someRandom)@aol.com

This was really annoying me. 

I exhausted all of my usual Spam hunting tactics, using things like Maldet and ClamAV to look for known spam scrips and hacks and came up empty. 
So I started looking at the traffic to the server as a whole, looking for patterns. 
Eventually I started seeing that xmlrpc POSTs was really the only pattern happening to all sites. 

tail -f /usr/local/apache/domlogs/*.com | grep POST | grep xmlrpc

91.197.232.105 - - [07/Jan/2017:06:15:55 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
91.197.232.105 - - [07/Jan/2017:06:16:53 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
91.197.232.105 - - [07/Jan/2017:06:17:04 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
91.197.232.105 - - [07/Jan/2017:06:17:14 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
91.197.232.105 - - [07/Jan/2017:06:17:16 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.32 - - [07/Jan/2017:06:21:53 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.58 - - [07/Jan/2017:06:26:10 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.58 - - [07/Jan/2017:06:27:05 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.58 - - [07/Jan/2017:06:28:40 +0000] "POST /xmlrpc.php HTTP/1.0" 301 - "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.58 - - [07/Jan/2017:06:28:35 +0000] "POST /xmlrpc.php HTTP/1.0" 200 384 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
212.47.230.108 - - [07/Jan/2017:06:28:59 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
212.47.230.108 - - [07/Jan/2017:06:30:43 +0000] "POST /xmlrpc.php HTTP/1.0" 200 384 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
212.47.230.108 - - [07/Jan/2017:06:31:15 +0000] "POST /xmlrpc.php HTTP/1.0" 301 - "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
51.15.43.32 - - [07/Jan/2017:06:31:40 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.144.12.15 - - [07/Jan/2017:06:31:47 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; uk; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13"
163.172.132.253 - - [07/Jan/2017:06:31:44 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"

So I started comparing the POSTs to the info@ spam in the Queue
Tossed this into the server command line 

sed -i '1i <Files xmlrpc.php>\norder deny,allow\ndeny from all\n</Files>\n ' /home/*/public_html/.htaccess

And now every domain has a block  that kills access to xmlrpc.php, and prevents this ongoing spam ordeal for this customer. 

FUN!!

There is no specific feature in WHM/cPanel to just block a complete domain from being able to send mail, so I used the following process:

Created a new file called blockeddomains with the domain that I wanted to block 

echo "domain.com" > /etc/blockeddomains

Now setting the proper permissions / ownership for the file:

chown root.mail /etc/blockeddomains && chmod 640 /etc/blockeddomains

Now I have to tell EXIM to look at this file, and use it to deny email from the domains listed in it. 

Open WHM, and Browse to WHM >> Service Configuration >> EXIM Configuration Manager >> Advanced Editor
Scroll down to the Add additional configuration setting button, and click on it, and add the following: 

domainlist blocked_domains = lsearch;/etc/blockeddomains

Scroll down further to the ROUTERSTART section and add the following:

reject_domains:

driver = redirect
domains = +blocked_domains
allow_fail
data = :fail: SPAM Source rejected: $domain is manually blacklisted.

Now you can block/unblock any domains simply by editing the /etc/blockeddomains file.

The problem is that it gets fucking hacked all the time, and abused to send billions of spam around the world.
Don't be an idiot and let yours get abused.

Securing wp-includes

A second layer of protection can be added where scripts are generally not intended to be accessed by any user. One way to do that is to block those scripts using mod_rewrite in the .htaccess file. Note: to ensure the code below is not overwritten by WordPress, place it outside the # BEGIN WordPress and # END WordPress tags in the .htaccess file. WordPress can overwrite anything between these tags.

# Block the include-only files.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^wp-admin/includes/ - [F,L]
RewriteRule !^wp-includes/ - [S=3]
RewriteRule ^wp-includes/[^/]+\.php$ - [F,L]
RewriteRule ^wp-includes/js/tinymce/langs/.+\.php - [F,L]
RewriteRule ^wp-includes/theme-compat/ - [F,L]
</IfModule>

# BEGIN WordPress

Note that this won't work well on Multisite, as RewriteRule ^wp-includes/[^/]+\.php$ - [F,L] would prevent the ms-files.php file from generating images. Omitting that line will allow the code to work, but offers less security.

Securing wp-config.php

You can move the wp-config.php file to the directory above your WordPress install. This means for a site installed in the root of your webspace, you can store wp-config.php outside the web-root folder.

Note that wp-config.php can be stored ONE directory level above the WordPress (where wp-includes resides) installation. Also, make sure that only you (and the web server) can read this file (it generally means a 400 or 440 permission).

If you use a server with .htaccess, you can put this in that file (at the very top) to deny access to anyone surfing for it:

<files wp-config.php>
order allow,deny
deny from all
</files>

Securing uploads

If file uploads are enabled, people can upload, and execute any arbitrary code, and use this to gain access to unintended areas of your site, or generate spam.
Creating a .htaccess file in the uploads folder with the following will prevent that.

<Files *.php>
deny from all
</Files>

Disable File Editing

The WordPress Dashboard by default allows administrators to edit PHP files, such as plugin and theme files. This is often the first tool an attacker will use if able to login, since it allows code execution. WordPress has a constant to disable editing from Dashboard. Placing this line in wp-config.php is equivalent to removing the 'edit_themes', 'edit_plugins' and 'edit_files' capabilities of all users:

define('DISALLOW_FILE_EDIT', true);

This will not prevent an attacker from uploading malicious files to your site, but might stop some attacks.

These are a few good places to start in securing your WP install. Of course keeping all updates applied to WordPress Core and any Plugins and Themes is very important as well. 

One of the easiest ways around this request is just to Inject a new Admin User, Do what you need to do, then delete it. 

INSERT INTO `user_db`.`wp_users` (`ID`, `user_login`, `user_pass`, `user_nicename`, `user_email`, `user_url`, `user_registered`, `user_activation_key`, `user_status`, `display_name`) VALUES ('1337', 'techsupport', MD5('password'), 'Technical Support', 'tech@thecrimsonhorror.com', '', '0000-00-00 00:00:00', '', '0', 'Tech Support');
INSERT INTO `user_db`.`wp_usermeta` (`umeta_id`, `user_id`, `meta_key`, `meta_value`) VALUES (NULL, '1337', 'wp_capabilities', 'a:1:{s:13:"administrator";s:1:"1";}'), (NULL, '1337', 'wp_user_level', '10');

The obvious changes to this would be depending on the specifics of the Database at hand.  After the INSERT INTO you would modify the user_db to be the actual Database name 
The next parts are the wp_users and wp_usermeta These would only change if the customer has modified their Database Prefix (If this is the case, then the wp_capabilities and wp_user_level would also have the wp_ replaced with the Database Prefix)  Modify the Username/Password and email address as needed and just run as an SQL Statement in phpMyAdmin. 

After you're done doing the testing, you want to clean up your User so it's not another entry point for hackers

Of course, the same things for this, change the Database name and Table Prefix to match the database you're working with, and all will be good. 
The UserID set to 1337 is an easy way to ensure that there are no conflicts, it's very unlikely that a user would have that many WP users.  

He moved from newsite.<domain> to <domain> and didn't make any updates in the WP Admin to account for the move to the new URL as per the Moving WordPress article on Codex. 

Figured I would repair the DB manually to get rid of all the 'newsite.' references. 

There were over 1600 references to the site name in the DB

Figured I would write up a little SQL Script that would make the update:

UPDATE wpjc_posts set guid=REPLACE(guid, 'newsite.customerdomain.com', 'customerdomain.com')
UPDATE wpjc_posts set post_content=REPLACE(post_content, 'newsite.customerdomain.com', 'customerdomain.com')
UPDATE wpjc_postmeta set meta_value=REPLACE(meta_value, 'newsite.customerdomain.com', 'customerdomain.com')
UPDATE wpjc_options set option_value=REPLACE(option_value, 'newsite.customerdomain.com', 'customerdomain.com')

Running this, updated 2000+ records referring to the old URL in a few seconds, and was literally the quickest resolution to this issue. 

Apparently he had this domain with us before, and transferred it over to GoDaddy for hosting, then was bringing it back. 

I've seen this happen a few times when the IP address they were pointing to was incorrect, or the httpd.conf was pointing to the home folder or IP incorrectly. 

I tried the basics like rebuilding the httpdconf and still getting issues. 

Had a look at the Zone file for the domain, and saw that on our side, the authoratative nameserver were set to his previous host. 
Rebuilt the Zone file, and it appears that the site started working. 

Probably something I should have thought of earlier, but at least I got to the bottom of it. 

Today, I'll cover a few Tweaks that can be done to Apache and MySQL to get that little extra bang out of your server that's not just a little VPS running 4-5 sites. 

For Easy Apache, I went with the following, to keep up with new features, but not be 'bleeding edge'

In the first stage we run the Easy Apache and selected the following:

Apache Version 2.4
PHP Version 5.5
 In step 5 “Exhaustive Options List” selected – Deflate – Expires – MPM Prefork and – MPM Worker

Next item to review is the Apache Global Configuration limits
WHM » Service Configuration » Apache Configuration » “Global Configuration”
These are pretty generic numbers based on how many GB's of ram on the server, these can still be tweaked a little further based on your usage, 

Apache Directive 	 	2GB             6GB             12GB 	 	

StartServers 	 	 	4 	 	8 	 	16 	
MinSpareServers 	 	4 	 	8 	 	16 	
MaxSpareServers 	 	8 	 	16 	 	32 	
ServerLimit 	 	 	64 	 	128 	 	256 	
MaxRequestWorkers 	 	50 	 	120 	 	250 	
MaxConnectionsPerChild 	 	1000 	 	2500 	 	5000 
Keep-Alive			On		On		On
Keep-Alive Timeout	 	5	 	5	 	 5
Max Keep-Alive Requests		50	 	120	 	120
Timeout				30		60		60


From here, I wanted to add in some basic Caching to Apache by default, so even is users don't specify Cache lifetimes, this will override.
WHM » Service Configuration » Apache Configuration » Include Editor » “Pre VirtualHost Include” 

# Cache Control Settings for one hour cache
<FilesMatch ".(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
Header set Cache-Control "max-age=86400, public"
</FilesMatch>

<FilesMatch ".(xml|txt)$">
Header set Cache-Control "max-age=86400, public, must-revalidate"
</FilesMatch>

<FilesMatch ".(html|htm)$">
Header set Cache-Control "max-age=3600, must-revalidate"
</FilesMatch>

# Mod Deflate performs data compression
<IfModule mod_deflate.c>
<FilesMatch ".(js|css|html|php|xml|jpg|png|gif)$">
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch bMSIE no-gzip
</FilesMatch>
</IfModule>

With this, basic media files, such as images and flash vids are forcibly browser cached for a day, and html files for an hour. 
This helps to account for the inaction of clients, and ensure that basic caches are in place. Making page loads a little faster, and server loads a little slower for repeat visitors. 



In /etc/my.cnf I made the following additions to optimize the connections and buffering. 

[mysqld]
local-infile=0
max_connections = 600
max_user_connections=1000
key_buffer_size = 512M
myisam_sort_buffer_size = 64M
read_buffer_size = 1M
table_open_cache = 5000
thread_cache_size = 384
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 256M
max_heap_table_size = 128M
max_allowed_packet = 64M
net_buffer_length = 16384
max_connect_errors = 10
concurrent_insert = 2
read_rnd_buffer_size = 786432
bulk_insert_buffer_size = 8M
query_cache_limit = 5M
query_cache_size = 128M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65535
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
max_write_lock_count = 8
slow_query_log
log-error
external-locking=FALSE
open_files_limit=50000

[mysqld_safe]

[mysqldump]
quick
max_allowed_packet = 16M

[isamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M

[myisamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M

#### Per connection configuration ####
sort_buffer_size = 1M
join_buffer_size = 1M
thread_stack = 192K

Then ran a Repair & optimize on the databases and restarted MySQL:

mysqlcheck --check --auto-repair --all-databases
mysqlcheck --optimize --all-databases
/etc/init.d/mysql restart

Here is how I had learned to combat this, by preventing my servers from sending outgoing spoofed mail, I know that I'm ensure that my domains are not part of the problem. 

By taking advantage of the EXIM Configuration Editor, we can effectively stop outgoing spoofing. 

I. Blocking all un-authenticated spoofed outbound emails

1. Login to WHM >> EXIM CONFIGURATION MANAGER >> ADVANCED EDITOR

2. Add the following entry in the top using Add additional configuration setting:
domainlist remote_domains = lsearch;/etc/remotedomains

 
3. Add the following code under acl_not_smtp >> custom_begin_outgoing_notsmtp_checkall:

deny
condition = ${if ! match_domain{${domain:${address:$h_From:}}}{ +local_domains : +remote_domains}}
message = Sorry, you don't have \
permission to send email from this server with a header that \
states the email is from ${lc:${domain:${address:$h_from:}}}.
accept

 
Here, the ACL will check for the presence of domain name part of the from address in either of the files – /etc/localdomains or /etc/remotedomains. If there is a mismatch, server will reject the email.

II. Blocking all authenticated spoofed outbound emails

1. WHM >> EXIM CONFIGURATION MANAGER >> ADVANCED EXIM EDITOR

2. Search for acl_smtp_data >> custom_begin_outgoing_smtp_checkall and add the following lines under it:

deny
authenticated = *
condition = ${if or {{ !eqi{$authenticated_id} {$sender_address} } \
{ !eqi{$authenticated_id} {${address:$header_From:}} } \
} \
}
message = Your FROM address ( $sender_address , $header_From ) 
must match your authenticated email user ( $authenticated_id ).
Treating this as a spoofed email.

 
Here, for all authenticated users, the rule will check whether the authenticated userid matches with the from address. If it matches, it will allow the email. Else, it will display the message “Your FROM must match your authenticated email user. Treating this as spoofed email”

PS: If the acl_smtp_data is mentioned as something else(like acl_smtp_data = check_message), locate check_message and add the above lines just under it.

IMPORTANT points to keep in mind

a. POP before SMTP won’t work with this setting. You will have to ask your customers to use the option – “My Server Requires Authentication” in the SMTP settings of their email client.
b. Username in the format user+domain.com will not work. They have to use user@domain.com instead.

These solutions have been tested on my personal cPanel server, and in a limited set of production servers. We have found it to be working in 100% of cases. However, using the above solution should be at your own risk. If you do not understand the ACLs posted above, always ask for expert opinion.