Blog.Photobucket.com

Some Users...

This maintenance has now ended and service is back to normal. If you are having problems accessing the site, please email support@photobucket.com

 --

Today (July 12th, 2011), at 5am MDT (UTC -6 hours), some users will notice some difficulties logging in to their Photobucket account. This is a scheduled maintenance, and everything should be back to normal within a few hours.

Thanks for your patience.

-The Photobucket Team

--

Hardware Issue Resolution

Hello Photobucket Users,

As many of you know, Photobucket experienced a hardware failure last week and some of you were not able to access your accounts or serve photos and videos to your sites. I know that this has been very frustrating. I sincerely apologize to each of you who were affected by this outage. I am happy to report that all accounts are now back online. 

Last Monday evening, we experienced a hardware failure; the odds of such a failure are over 1 in a billion. Throughout this time, we worked closely with our storage and data management partner to make sure that all measures were taken to preserve and restore all data. No data was lost, but the recovery process took much longer than expected. Going forward, we will be adding an additional layer of redundancy as an added measure of protection.

Throughout the last two weeks, the Photobucket team was inspired by your comments and patience. We love your passion for Photobucket and we thank you. I would also like to thank the Photobucket team that literally worked day and night to resolve this issue.

Thank you,

Tom Munro
CEO

Hardware Issue Update

Photobucket Users,

For those of you who are still affected by the current hardware failure, we’d like to let you know that we’re making good progress on the resolution. The complexities are still hindering us from making an accurate time-line for the completion of the solution. However, we are optimistic that tomorrow’s update will contain more information and give us a better idea on the completion of the solution.

Thanks for your patience,

Jim Goss

VP of Operations

Hardware Issue Update

Hello Photobucket Users,

We’ve worked through the weekend with our infrastructure vendors on the issue that’s still affecting some of our users. Despite the many layers of redundancy that Photobucket utilizes in our data centers, we experienced a rare multiple hardware failure.

The probability of such an event is extremely rare (1 in 1 billion).  Since it is so rare, we have had a difficult time estimating a resolution.  We should have a resolution or at least a better timeline for resolving the issue by tomorrow (4/19/11) or Wednesday (4/20/11). At that time, we will post here, as well as our Facebook page and on Twitter.

We do understand your frustrations and there is a large team of Photobucket employees, consultants and vendors working hard day and night to resolve this issue as quickly as possible.

Yours,

Jim Goss

VP of Operations

Maintenance Update

Hi Photobucket users,

I would like to extend my thanks to each of you who are still affected by the maintenance outage for your patience. Since my last post, our team has restored service to about a quarter of the remaining affected users - leaving less than 0.2% of users still experiencing this problem - and we continue to work day and night to get everyone back online.

We at Photobucket are very focused on resolving this error as soon as possible, and do acknowledge that those of you who are in the last remaining group are frustrated. We are continuing the extended diagnostic and recovery process, and will not rest until we are finished. The complexity of this process is unfortunately making a definitive end time impossible to accurately estimate.

Many of you have reached out to us through email, Facebook and Twitter, and we thank you for your feedback. Please be secure in the knowledge that our team is working persistently on this matter.

Yours,

Jim Goss

VP of Operation

Maintenance Update

Photobucket customers,

Thank you all for your continued patience while we work persistently on this issue. While this is still affecting only a very small percentage of our users, we realize it is a big issue for those impacted. We’d like to give you a current update as we continue to work night and day on resolving this issue.

In order to clarify and dispel some concerns, let me assure you that Photobucket has not been hacked, but instead we are working through a hardware issue. Right now, we’re continuing the process of going through an extended diagnostic and recovery process.

We understand your frustrations, and promise you that we are doing everything we can to bring everything back on line as quickly as possible. However, we do feel it best to advise the affected users that this process is highly unusual and time consuming.

Thank you for your patience, and for all your feedback.

Yours,

Jim Goss

VP of Operations

Maintenance Update

Photobucket customers,

Let me begin by once again expressing my apologies for the issues you have been experiencing, and for the delay in restoring service. We continue to work diligently to resolve the problems and would like to keep you up-to-date on how things are going.  The vast majority of you that were impacted should now or very soon notice that your accounts and images are back up again. However, not every user has been restored to service; we still are working to restore less than 1% (0.25%) of our user base.  If your account is still not working, please be assured that we know about it and continue to work toward fully restoring service for all users as quickly as possible. We will continue to provide updates as they occur.

We hear your frustration and appreciate all of your feedback.

Thank you,

Jim Goss
VP of Operations

New Maintenance Update

Photobucket customers,

We again apologize for the issues some of you are experiencing, and for not having fully resolved the issue as of yet. We want to make sure we are keeping you up-to-date on what is happening and what we are doing to fix the problem as quickly as possible.

We experienced a problem with a portion of our infrastructure, which affected the ability for some users to log in and access their account. We immediately began maintenance to fix the problem, and have been working diligently to get things running again as quickly as possible. However, due to the sheer number of files and the enormous amount of data involved, this is taking a significant amount of time and much longer than we initially anticipated.

Please know that we are doing everything possible to restore service as quickly as possible. We understand your frustration and we will continue to update you as we get better estimates around a completion time.

Thank you for your patience,

Jim Goss

VP of Operations

Maintenance Update

Hey Photobucket Fans,

We’re still experiencing some maintenance issues which affect a small percentage of our users. Affected users won’t be able to log in, and all link images will be replaced with the following:

Rest assured your photos are safe, and we’re working diligently on getting this fixed ASAP.

Right now, the estimated time to verify all data is approximately after midnight today (04/12/11), 12am MDT (-6 GMT). We apologize sincerely for the error, and we understand the problems this creates.

Thank you for your patience, and we’ll keep you updated through our Facebook (http://www.facebook.com/photobucket) and Twitter pages (http://twitter.com/Photobucket).

Thanks,

The Photobucket Team.

Site Maintenance Tonight at Midnight

Attention Photobucket fans:

The site will be down tonight (March 3rd, 2011) from 12 midnight to 1:30am MST for maintenance. Here's some relevant information for you all:

1. The website and content moderation will be unavailable during this time.
2. Direct links will continue to be served.

Thanks for your patience!

Site Issue Resolved: March 21st

Status as of 12:30PM Pacific Time: March 21st, 2007: During the night on March 20th 2007, a small percentage of Photobucket users experienced issues accessing their content.

The moment we were alerted to the issue, our special forces team worked through the night, and now the issue has been resolved. 

No user should have experienced any data loss. There is no need to either make a new account, or re-upload your content.

Please know that we place the very highest priority on the reliability of our service. We fully appreciate that for those people who experienced an inaccessible account, this was frustrating and extremely inconvenient. Despite the volume of comments here, the number of users affected was only a small percentage of our overall user base. And every customer matters to Photobucket.

We truly appreciate your patience and thank you for choosing Photobucket.

If you have any specific questions, please email us at pbfeedback@photobucket.com.

Some albums temporarily unavailable

One of our master filesystems had a disk fail. We do have raid so no data is lost, however it was slowing things down to the point where everyone was effected.

We have taken the albums on this volume offline to speed things up. It should be finished within an hour.

Sorry

Alex

Update 3:30 mst (82% rebuilt) so expect within the 1/2 hour

Update 4:00 mst - DONE! Sorry about that :( All is back to normal

Little issue early Wednesday morning

Our network provider was doing a non service affecting maintenance at 4am when things went south. They had both their engineers and the NOC involved, things were resolved as quickly as possible.

The maintenance was to add a redundant supervisor card (or the brain of a Cisco Router) to ensure future uptime and availability. Ironically it caused an outage in doing so, sorry.

-alex