Site Migration - 3-5 Sep 10

[titus]

Hello,

A site migration is planned to take place over the weekend of 3 - 5 Sep 10. Part of a tech refresh project to facilitate growth of planned software releases and site traffic, the whole site will be migrated off to a different data center to an entirely new infrastructure platform.

The site will be closed beginning at 03:00 EST 3 Sep 10 and if all goes well, be opened by 12:00 EST 3 Sep 10. However, since a DNS change is involved, it may take up to a day or two for you to reach to the new servers.

Testing has been in place for some time now and everything looks stable. If you experiences any problems by Saturday 4 Sep 10, please send an e-mail to support@canreef.com.

Titus

[robert]

Titus - change the TTL to something short like couple of hours couple days before the swap if you can. It will expedite the DNS propagation and limit the downtime for users.

Good luck with the transfer.

Cheers!

[mark]

thanks for the heads up, I've gotten a little anxious before on those rare times the sites gone down (weird)

[StirCrazy]

which software you changing to Titus?

Steve

[titus]

Titus - change the TTL to something short like couple of hours couple days before the swap if you can. It will expedite the DNS propagation and limit the downtime for users.
Thanks. Am thinking about that also.

thanks for the heads up, I've gotten a little anxious before on those rare times the sites gone down (weird)
Yes it's time due for an upgrade. We'll also be implementing a separate hardware to do proper monitoring for the just in case. Overall platform has a much higher capacity. There will also be multiple redundancy for backups to keep the data safe.

which software you changing to Titus?
Upgrade on vBulletin's software. And a few other items mainly to add functionality and customizations. Unfortunately the infrastructure aspect has taken much longer than initially planned because everything is new and the scope is also much broader than the typical community sites. As such, I don't anticipate to be able to do all the software releases by end of year. This is a long term plan.

Titus

[titus]

Hello,

It appears that the site migration was successful. In fact, there was a DoS attack on the old data center on September 3, 2010 between 17:35 ET and 17:45 ET. Glad we were out of there before that happens.

Please let me know if you encounter any issues.

Titus

[MitchM]

I tried a FTOTM link that doesn't seem to work:

http://canreef.com/ftotm/nov08/index.php

Mitch

[Mandosh]

Banner links don't look like they're working yet

[titus]

Hello,

Thanks. Just tried the FTOTM link and it's working. The banners from what I understand have always worked. Just tried to reload the page and clicked on them. All appears to be in order. I can't see why both the FTOTM and banner links would fail as both the new and old site are up in parallel so regardless of DNS issues you should be able to retrieve something.

Anyway, can you guys try again?

There was a mailing issue however. Notification e-mails were not being delivered. They're now being delivered but I'm seeing rejections from a few mail servers stating that my IP has a poor reputation. I suspect this is because there isn't a PTR record for the mail server. Need to fix this.

Mails to titus@canreef.com and support@canreef.com are still down also.

Titus

[reefwars]

banner ads dont work for me either i just tried:)

[titus]

Hello,

Can you please kindly let me know if it's the graphics not displaying, or when you clicked on it it doesn't direct you to the vendor's site? If the graphics is not displaying, can you right click on it and see what file it is attempting to load? For Firefox, it's right click -> Copy Image Location. For Internet Explorer, it's right click -> Properties -> Address.

I'm also seeing a bunch of mail bounce back. Some are because the member's e-mail is not valid. Some are because our IP is being blocked. If you have a shaw.ca address, then you'll most likely not able to receive e-mails from the site. Shaw.ca believes our mail server has a poor reputation.

Titus

[MitchM]

If I click on a banner ad, I get a "...because the server unexpectedly dropped the connection..." message.

If I click on the FTOTM link, I get a "...because Safari can't find the server "canreef.com" message.

For either error message if I replace canreef.com with 216.187.96.54 in the address bar, they both work.

I do use shaw.ca, so I guess I'll change that.

Mitch

[titus]

Hello,

Mitch, thanks and that's very interesting. Can you do the following:
1) Start -> Run, type Cmd, press Enter
2) Type "nslookup www.canreef.com", press Enter
You should see it resolving to 96.31.86.198
3) Type "nslookup mail.canreef.com", press Enter
You should see it resolving to 96.31.86.198 also
4) Type "nslookup ns1.canreef.com", press Enter
You should see it resolving to 96.31.86.200
5) Type "nslookup ns2.canreef.com", press Enter
You should see it resolving to 96.31.86.201

If the above doesn't return the proper info, do:
Type "ipconfig /flushdns", press Enter

Titus

[MitchM]

I'm on a Mac using Terminal, so I'm not sure about that last command, but this is what I get:


WV-MacPro-2:~ mitchandbev$ nslookup www.canreef.com
Server: 64.71.255.253
Address: 64.71.255.253#53

Non-authoritative answer:
Name: www.canreef.com
Address: 96.31.86.198

WV-MacPro-2:~ mitchandbev$ nslookup mail.canreef.com
Server: 64.71.255.253
Address: 64.71.255.253#53

Name: mail.canreef.com
Address: 96.31.86.198

WV-MacPro-2:~ mitchandbev$ nslookup ns1.canreef.com
Server: 64.71.255.198
Address: 64.71.255.198#53

Non-authoritative answer:
Name: ns1.canreef.com
Address: 96.31.86.200

WV-MacPro-2:~ mitchandbev$ nslookup ns2.canreef.com
Server: 64.71.255.198
Address: 64.71.255.198#53

Non-authoritative answer:
Name: ns2.canreef.com
Address: 96.31.86.201

WV-MacPro-2:~ mitchandbev$ ipconfig /flushdns
usage: ipconfig <command> <args>
where <command> is one of waitall, getifaddr, ifcount, getoption, getpacket, set, setverbose

[titus]

Hello,

Okay so the DNS is working properly on your end, and there's no need to flush your computer's DNS entries.

I searched online for the two error messages you saw, and found the following:
http://forums.macrumors.com/showthread.php?t=380238
http://discussions.info.apple.com/message.jspa?messageID=10324914

Can you try to use Firefox or IE instead of Safari?

Titus

[MitchM]

Firefox Macintosh gives me "cannot find server" for TOTM
Firefox Macintosh gives me "The connection to the server was reset while the page was loading"

Firefox Windows gives me a blank page for a banner ad
Firefox Windows gives me "cannot find server" for TOTM

Internet Explorer Windows gives me "Internet Explorer cannot display the webpage" for both issues.

From the cmd window, Windows gives me this:

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Macpro win7>nslookup www.canreef.com
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 64.71.255.198

Non-authoritative answer:
Name: www.canreef.com
Address: 96.31.86.198


C:\Users\Macpro win7>nslookup mail.canreef.com
Server: dns.rnc.net.cable.rogers.com
Address: 64.71.255.198

Name: mail.canreef.com
Address: 96.31.86.198


C:\Users\Macpro win7>nslookup ns1.canreef.com
Server: dns.rnc.net.cable.rogers.com
Address: 64.71.255.198

Non-authoritative answer:
Name: ns1.canreef.com
Address: 96.31.86.200


C:\Users\Macpro win7>nslookup ns2.canreef.com
Server: dns.rnc.net.cable.rogers.com
Address: 64.71.255.198

Non-authoritative answer:
Name: ns2.canreef.com
Address: 96.31.86.201


C:\Users\Macpro win7>

[titus]

Hello,

Mitch, thanks for the info. Unfortunately I am unable to determine a cause for this.

Anyone has any ideas?

Titus

[MitchM]

Hello,

Mitch, thanks for the info. Unfortunately I am unable to determine a cause for this.

Titus

It's the least I can do. I really appreciate the effort it takes for everyone to keep this board running. There is a lot of valuable information here.
There are always workarounds.:smile:

Mitch

[fishoholic]

I just clicked on a banner and I got a "Problem loading page" message:

The connection to the server was reset while the page was loading.


* The site could be temporarily unavailable or too busy. Try again in a few
moments.

* If you are unable to load any pages, check your computer's network
connection.

* If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

Also when I click on TOTM I just get the write up but no pictures show. This funny message appears at the top of the page as well:

Highslide JS"; } } ?>

[PoonTang]

Also when I click on TOTM I just get the write up but no pictures show. This funny message appears at the top of the page as well:

Highslide JS"; } } ?>

+1

[titus]

Hello,

Can you right click on it and see what address the links are attempting to load? For Firefox, it's right click -> Copy Image Location. For Internet Explorer, it's right click -> Properties -> Address.

Titus

[PoonTang]

Hello,

Can you right click on it and see what address the links are attempting to load? For Firefox, it's right click -> Copy Image Location. For Internet Explorer, it's right click -> Properties -> Address.

Titus

http://www.canreef.com/ftotm/june10/images/fts1/thumbs/DSC_3272.JPG
but if you click on the image it takes you here:
http://www.canreef.com/ftotm/june10/

[titus]

Hello,

Okay I can see that too. I don't know what it is but I have some ideas. Let me fix it later today.

Titus

[titus]

Hello,

Both issues fixed. FTOTM was due to new PHP version being non 100% code compatible with the old version. Top banner link was due to firewall setting.

Titus

[titus]

Hello,

Once again, Shaw is rejecting e-mails from Canreef whenever we send out notifications for new PMs or replies to threads. Previously, I have received the following response from them on my inquiry:
Hello,

The error message indicates that the email server at the IP address 96.31.86.198 had been blocked from sending mail to our email server, as the sending server had been flagged by our spam protection system has having sent out a high number of suspicious emails. At this time, our server is not blocking that IP address, or restricting access.

Generally speaking these sorts of issues will correct themselves within a few hours. If the situation persists for longer than that, please contact your mail administrator for further assistance.

If there is anything else we can help you with, please feel free to contact us by phone at 604-629-3000, email or live chat (https://secure.shaw.ca/apps/secure/livechat.aspx?Portal=shaw-chat-nan).


Thanks,

Dave


However due to the nature of the e-mails being sent out, Shaw may have flagged us as spamming again. I have e-mailed them again by responding to the above e-mail at van.help@sjrb.ca.

If you are registered with a Shaw address, I suggest either changing it to another one, or call Shaw and let them know that they should adjust their spam setting.

Titus

[ALang]

HI, Still have no email from shaw. I phoned them to repeat what you posted, any all I got was," if they can't do anything about this, you certainly won't be able to." And they were totally baffled as to what I was trying to convey to them.

So still no auto-notifications, pms, anything...
Wish this could be rectified without us haveing to set up yet another email account.

[titus]

Hello,

Yeah I know. Each day I'm getting reject e-mails coming into my personal inbox. I stopped checking for 2 days (on biz trip at the moment) and I just saw close to 600 reject e-mails!

I was referred to their network security team, which I'd approached, and provided samples of one of the reject e-mail's content but was given a standard answer. I inquired if they can put an exception on the firewall and have heard of nothing. If you can call, please ask to speak to this Morgan Pierce person explaining how much trouble we're all having.

Apologies for not posting this earlier. Thought I'd chase them up again but have been seriously ill.

Titus


Subject: RE: Shaw Email Administration Team - Auto Reply
Date: Tue, 14 Sep 2010 16:11:50 -0600
From: socmail-admins@sjrb.ca

Hi Titus,

Could you have your forum MTA send a message to pd7ims1@shaw.ca so that we can analyze what is happening here?

Please let us know at mail-admins@sjrb.ca when you have done so.

Thank-you,


Morgan Pierce
Internet Product Specialist
Shaw Communications Inc.Hello,

Just did it and have the following in my /var/log/maillog...
Sep 16 08:09:17 production postfix/pickup[32343]: D34914403B0: uid=0 from=<root>
Sep 16 08:09:17 production postfix/cleanup[1642]: D34914403B0: message-id=<20100916150917.D34914403B0@mail.canreef.com>
Sep 16 08:09:17 production postfix/qmgr[27631]: D34914403B0: from=<root@canreef.com>, size=402, nrcpt=1 (queue active)
Sep 16 08:09:18 production postfix/smtp[1644]: D34914403B0: host idcmail.shaw.ca[24.71.223.11] refused to talk to me: 554-idcmail.shaw.ca 554 Your connection from 96.31.86.198 has been rejected due to poor reputation.
Sep 16 08:09:18 production postfix/smtp[1644]: D34914403B0: host idcmail.shaw.ca[64.59.134.8] refused to talk to me: 554-idcmail.shaw.ca 554 Your connection from 96.31.86.198 has been rejected due to poor reputation.
Sep 16 08:09:18 production postfix/smtp[1644]: D34914403B0: host idcmail-mx1so.cg.shawcable.net[24.71.223.11] refused to talk to me: 554-idcmail.shaw.ca 554 Your connection from 96.31.86.198 has been rejected due to poor reputation.
Sep 16 08:09:18 production postfix/smtp[1644]: D34914403B0: to=<pd7ims1@shaw.ca>, relay=idcmail-mx2no.cg.shawcable.net[64.59.134.8]:25, delay=0.95, delays=0.22/0.01/0.72/0, dsn=4.0.0, status=deferred (host idcmail-mx2no.cg.shawcable.net[64.59.134.8] refused to talk to me: 554-idcmail.shaw.ca 554 Your connection from 96.31.86.198 has been rejected due to poor reputation.)


Command I executed was:
echo "E-mail test from mail.canreef.com at 96.31.86.198 as requested by Morgan Pierce" | mailx -s "ATTN: Morgan Pierce for mail.canreef.com" pd7ims1@shaw.ca

Regards,
TitusHi Titus,

The reason you are seeing this is due to the reputation of the IP address that mail is being sent from.

If you reference senderbase.org you will see the reputation there: http://www.senderbase.org/senderbase_queries/detailip?search_string=96.31.86.198

Once that reputation improves, sending to shaw.ca addresses will resume automatically.

Thank-you,


Morgan Pierce
Internet Product Specialist
Shaw Communications Inc.Hello,

Well this is going to take some time as this is a new IP for the site.
Can we please have an exemption placed on the IP? If you check, all the mails being sent out are all valid, non-spam e-mails.

Regards,
Titus
Titus

[titus]

Hello,

I just want to highlight how ridiculous the situation is. Since I have moved Canreef to the new data center, I have received over 9,100 reject e-mails from Shaw to my personal mail box.

On average I receive over 100 of such e-mails a day.

Titus

[fishoholic]

Wow that's ridiculous :surprise:

[scherzo]

This is funny because Shaw's own IP range's reputation is poor.

Can't you just change the IP address? I'm pretty sure the new Data Centre can do it and have it switch over without us noticing.

I've done it with my sites a few times.... It is seamless if done right. The data centre should be able to setup both sites running simultaneously on two IPs with just the database linked (unless it is on the same server... don't answer this question out loud... ;) )

Do you have access to another mail server to send out the e-mails? That is another option.

[titus]

Hello,

In summary, this was most likely an error with Cisco, at least as far as I'm concerned. Finally I have been able to get in touch with their support via e-mail. For the record, I've checked again on Senderbase that my IP was still flagged as Poor prior to e-mailing them. As soon as I saw a response from them, claiming obviously that there's nothing with my IP, the same Senderbase site has changed to flag my IP to be Good. This was sometime within the week of 11 - 15 Apr 11.

Since then, I've stopped receiving the loads of rejects I get from Shaw daily. This must indicate that all the members with Shaw e-mail address are now getting e-mails.

Titus

[ALang]

Will give it a try again. Thanks Titus.