Hey all,
I spent the better half of 9 days transferring 200+GB worth of files to an Amazon S3 bucket via the transfer tool under "Manage Storage Services". All of the files transferred fine like expected, but now when selecting S3 as my default storage, files take an extremely long time to upload. When I revert back to my server as the storage, everything uploads quickly.
Using S3 as my default storage, a 1MB file will take a minute or so to upload, a 3MB maybe takes 3-5minutes, a 20MB file takes 5+ minutes to upload and so on. I love the idea of having this external storage for my site because my members upload a lot of content, but this slow upload using S3 is not a viable option.
So, is anybody else using S3 as their default storage? How is it working for you when uploading files? Slow? Fast? I'd love to get some input on this.
I think it depends on the region you selected. I recall seeing something about this for someone else. https://forums.aws.amazon.com/thread.jspa?threadID=116513 try a traceroute and there's one mentioned by Amazon in that thread. See if it shows any issues. We use Amazon here and at our demo and hadn't noticed an issue.
Hi Donna,
Thanks for the response.
I am located in Dallas, my server is located in Chicago and my bucket is located in N.California. Maybe moving my files over to a bucket closer to my server (Ohio?) would improve a tiny bit? I'll review that thread in the link you provided.
It should also be noted that I'm using SE 4.8.12 on PHP 5.6 (apache 3). Not sure if these outdated versions are adding to the problem but I won't be positive until Saturday when I upgrade to PHP 7.x (apache 4) and the latest version of social engine/plugins. I'll follow up to this thread once everything is upgraded.
But in the mean time, you said the social engine demo uses S3 as the default storage location? Because browsing around the demo and viewing source on profile pictures, uploaded music etc. shows the files to be uploaded locally.
Example:
Profile photo: https://demo.socialengine.com/profile/gary --> https://demo.socialengine.com/public/user/1a/01/f79e96a5e722ebe1c115e678c8fc1528.jpg
Music: https://demo.socialengine.com/music/2/travelin-039-man --> https://demo.socialengine.com/public/music_song/35/01/b59af033c23a2ba8bb3d330eda39677e.mp3
You should upgrade as there were a lot of improvements since that version. I'll double check the demo as I thought it was using S3 like the rest of our sites.
Ok yeah it uses Amazon but since it gets reset, the main files in there for the demo content are stored elsewhere due to the reset each day. Newly uploaded stuff for people testing should be on Amazon if you test by signing up and loading a photo or something. This site also uses Amazon, just double checked.
Thanks for the feedback, Donna. I've upgraded to the latest version of SE4 as well as PHP. I will update you all how S3 upload speeds are in a few days, now that I've upgraded.
You can actually go edit a core SE file to make the transfer go a lot faster in the future. For some reason SE will only transfer a few files at one time but you can edit the limit for the transfer to make it send more at one time.
"/application/modules/Storage/Plugin/Job/Transfer.php"
If you look in there, you can change it for future transfers. Just do not use too large of a number or the transfer may fail or timeout.
Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.
Casey said:
"/application/modules/Storage/Plugin/Job/Transfer.php"
If you look in there, you can change it for future transfers.
Are you referring to this instance?:
$file = $filesTable->fetchRow($filesTable->select()
->where('service_id != ?', $service_id)
->where('file_id >= ?', (int) $position + 1)
->order('file_id ASC')
->limit(1));
Casey said:
Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.
Yeah maybe at one point I'll go that route. For 200GB+ I'm only paying roughly $10/month so it's not that bad for me. The issue is that my members upload a lot. Maybe I'll use a Host Gator shared account for the "unlimited" storage? What do you think?
Change line 33.... the limit is set to 5... change the 5 to whatever...try 100....
// Prepare
$service_id = $this->getParam('service_id');
$position = $this->getParam('position', 0);
$progress = $this->getParam('progress', 0);
$total = $this->getParam('total');
$limit = $this->getParam('limit', 5);
$isComplete = false;
$break = false;
Are your users multi-geographical? Or are they roughly in the same area/location?
oh...BTW.... this method is only for transferring ALL files between storage systems. If you want to move all files off local storage to AWS or to another 3rd party storage server or all files back to your local storage. It will not take 9 days in the future to make the changes. To add an answer to your question....S3 upload speeds are just slow for everyone. You need to look at S3 Transfer Acceleration. Not sure if it will fork with SE.
http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html
Use this tool to check differences
@OTAw, I edited your post to add code tags instead of deleting the code per our posted terms. Please do use the code tags as explained in our posting guidelines for any code you post. Thank you! Stickied guidelines for posting are at the top of the forum sections.
Great thread guys! Thanks @Casey for your input too.
Awesome post guys.
I am planning to implement AWS S3 as prescribed in this post.
The plan is to make S3 the final and only storage place for the site and use Transfer Acceleration.
Are there any updates on this strategy ? any new things to watch out for?
Thanks
I'm searching this thread, is there any detailed guide also on how to prepare the server on Amazon s3 side? Since I'm also interested in getting more servers around the globe.. to make it work faster..
Best regards, G
Did you search our Learning Center knowledebase? https://www.socialengine.com/support/article/5243168/se-php-how-to-use-the-cloud-storage-feature that tutorial should help some.
This link isn't working: https://www.socialengine.com/support/article/5243168/se-php-how-to-use-the-cloud-storage-feature
Is there a more recent link?
I am trying to transfer files to S3 and I am getting errors, so I need to know exactly how to set up the S3 side.
FYI, the Learning Center link in the menus above goes to the tutorials. The link you have is old. Here's the new one https://socialengine.atlassian.net/wiki/spaces/SU/pages/5243168/SE+PHP+-+How+to+use+the+Cloud+Storage+Feature . Just making you aware of where the new learning center is as well.
Thanks, but I have followed these instructions exactly and I am getting these unhelpful errors in the job log when the transfer job runs:
I was hoping for a bit more detail than "create an S3 account and get the security credentials and enter them into the storage service page"
I want to get all of the files in the public folder onto S3 storage so I can upgrade to SE5 because our hosting provider says that the hundreds of thousands of files in public are causing the upgrades to fail.