This page (set up just so I could see how Blogger works) is a portal
to our adventures in food and drink, recipes, disasters, triumphs...


Please use the links below or go directly to the real blog from OUR kitchen



` ` `


16 July 2011

about my real blog... yet again....

 

edit 17 July 05:23: And etherwork.net is back online. Don't you just love summer heatwaves? (Yesterday, at 5:00 it was 31C outside. The temperature went down to 22C overnight. Environment Canada is forecasting that it will go up to 34C today.)



Updates - Sunday, July 17th, 2011 5:07:21am

July 16, 11:27 PM - Murphy's Week

DNS DOS and now *another* chiller (A/C) failure.

The server locked up 3 times preventing data damage. We are bringing it up again and the NOC is working to get a new chiller unit brought in asap.



::::::::::::::::::::::::::::::

16 July 5:48 PM:

The webhost for my real blog on etherwork.net is offline AGAIN. I have no idea how long it has been offline, nor when it will return.

It could be the extreme heat here. Or it could be more of what was going on yesterday. Clearly, the kiddies don't have enough to keep them occupied. (Apparently there were problems yesterday.) Here is an excerpt from the updates page:



Updates - Saturday, July 16th, 2011 5:47:38pm

July 15, 9:08 AM - DOS Update
[M]ost things still working but still not 100%.

I got a call about 30 mins ago and informed that "changes were made" and things should be better if I could make a small change to our DNS servers. When I did, the sites were immediately loading faster, there was no delay in resolving DNS.

The DNS attack was very simple - someone (or group of people) were sending tons of DNS requests to DNS servers (ours, our colo's, our upstreams - Bell Canada, and other customers).

Normally there would be a total of 70,000 packets per section for *all* traffic - web, ftp, dns, everything else. At the peak yesterday, DNS alone was accounting for 108,000 packets per second on top of the normal volume.

That is like 108,000 different people loading up your web site each and every second. The network was so flooded that DNS could not get out and normal traffic flow was impeded.

Well it's over for now and we hope for good!

We are watching vigilantly.

July 15, 1:40 AM - Much Better

One of our upstreams found a lot of traffic coming in and hitting us and has filtered it - a bit aggressively but it is keeping the servers up.

They are going to keep looking into the the root cause and make a more intelligent filter. In the mean time, the majority of people can see all the sites.

July 15, 12:51 AM - DNS DOS
Posted by: tony
We have been hit with a large DNS DOS attack since about 10am Eastern Time. My ISP was also having DNS issues which has made this entire thing more difficult to work through.

The DOS is still going but not as strong as before.

We are working with upstream providers to identify what is going on and who is doing this. As soon as we have news we will advise.

The problem is that the source of the traffic has spoofed their address to places that would never DOS. Tracking this down is going to be tricky.




As always, the way to know whether etherwork is back online is to refresh this page. If you see a leafy image at the top of the page, then you can access the real blog.