Site times out

Sometimes when I post a message, the site takes too long to load and times out. It always works in the end, but I wondered why this happened from time to time?

We think it's because of heavy SE bot activity since the upgrade. However we don't want to impede it because the site needs to be re-indexed in the SE's

Ah well, at least it's not just me experiencing this! A necessary evil .....

No, I get it too. I'm working on (between other projects) adding a cookie based "lastPost" feature, so that the last typed post will be retrievable should the submit time out. Just tough to do a few minutes here and there between other things.

I know the feeling. It's getting to be a complete pain though :( it just did it again now when I tried to read this reply. Can't you temporarily alter robots.txt and keep the pesky things out?

Well that's the problem, they all seem to be known large SE bots (google, yahoo, jeeves, etc), and we need them here. It shouldn't last THAT long. They're just here because the indexed links they have are now all irrelevant with the new software. The whole site has to be re-indexed.

Makes me realise what a professional job Seventwentyfour do with my website which is checked and pinged every few minutes - but the whole thing is cleverly unobtrusive without any impact on the site. I know Google etc are a bit different of course. As you say, it ought to improve.

Hitting the back button usually will bring the page back with post in tact. Also, if you're making a huge post, you might want to copy it before you hit submit and then you can just paste it in.

I'm not sure if it is related but I'm also getting periodic timeouts hitting the front page, not just when posting a message. Seems like it was fine after the upgrade and just started happening in the last day or two.

No, it's been happening a bit longer than that. We feel it began when the bots started hitting us in droves, to re-index.