Amazon S3 as default storage?

    • 36 posts
    October 11, 2017 2:43 PM EDT

    Hey all,

     

    I spent the better half of 9 days transferring 200+GB worth of files to an Amazon S3 bucket via the transfer tool under "Manage Storage Services". All of the files transferred fine like expected, but now when selecting S3 as my default storage, files take an extremely long time to upload. When I revert back to my server as the storage, everything uploads quickly.

     

    Using S3 as my default storage, a 1MB file will take a minute or so to upload, a 3MB maybe takes 3-5minutes, a 20MB file takes 5+ minutes to upload and so on. I love the idea of having this external storage for my site because my members upload a lot of content, but this slow upload using S3 is not a viable option. 

     

    So, is anybody else using S3 as their default storage? How is it working for you when uploading files? Slow? Fast? I'd love to get some input on this.

    • Moderator
    • 6923 posts
    October 12, 2017 4:23 AM EDT

    I think it depends on the region you selected. I recall seeing something about this for someone else. https://forums.aws.amazon.com/thread.jspa?threadID=116513 try a traceroute and there's one mentioned by Amazon in that thread. See if it shows any issues. We use Amazon here and at our demo and hadn't noticed an issue.  

    • 36 posts
    October 12, 2017 9:09 AM EDT

    Hi Donna,

     

    Thanks for the response.

     

    I am located in Dallas, my server is located in Chicago and my bucket is located in N.California. Maybe moving my files over to a bucket closer to my server (Ohio?) would improve a tiny bit? I'll review that thread in the link you provided.

     

    It should also be noted that I'm using SE 4.8.12 on PHP 5.6 (apache 3). Not sure if these outdated versions are adding to the problem but I won't be positive until Saturday when I upgrade to PHP 7.x (apache 4) and the latest version of social engine/plugins. I'll follow up to this thread once everything is upgraded.

     

    But in the mean time, you said the social engine demo uses S3 as the default storage location? Because browsing around the demo and viewing source on profile pictures, uploaded music etc. shows the files to be uploaded locally. 

     

    Example:

    Profile photo: https://demo.socialengine.com/profile/gary  --> https://demo.socialengine.com/public/user/1a/01/f79e96a5e722ebe1c115e678c8fc1528.jpg

    Music: https://demo.socialengine.com/music/2/travelin-039-man  --> https://demo.socialengine.com/public/music_song/35/01/b59af033c23a2ba8bb3d330eda39677e.mp3

    • Moderator
    • 6923 posts
    October 12, 2017 9:20 AM EDT

    You should upgrade as there were a lot of improvements since that version. I'll double check the demo as I thought it was using S3 like the rest of our sites. 

    • Moderator
    • 6923 posts
    October 12, 2017 9:23 AM EDT

    Ok yeah it uses Amazon but since it gets reset, the main files in there for the demo content are stored elsewhere due to the reset each day. Newly uploaded stuff for people testing should be on Amazon if you test by signing up and loading a photo or something. This site also uses Amazon, just double checked.

    • 36 posts
    October 14, 2017 8:08 PM EDT

    Thanks for the feedback, Donna. I've upgraded to the latest version of SE4 as well as PHP. I will update you all how S3 upload speeds are in a few days, now that I've upgraded.

    • 66 posts
    October 23, 2017 11:42 PM EDT

    You can actually go edit a core SE file to make the transfer go a lot faster in the future. For some reason SE will only transfer a few files at one time but you can edit the limit for the transfer to make it send more at one time.

     

    "/application/modules/Storage/Plugin/Job/Transfer.php"

    If you look in there, you can change it for future transfers. Just do not use too large of a number or the transfer may fail or timeout.

     

    Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.


    This post was edited by Casey at October 23, 2017 11:44 PM EDT
    • 36 posts
    October 24, 2017 12:09 AM EDT

    Casey said:

     

    "/application/modules/Storage/Plugin/Job/Transfer.php"

    If you look in there, you can change it for future transfers.

     

     Are you referring to this instance?:

    $file = $filesTable->fetchRow($filesTable->select() ->where('service_id != ?', $service_id) ->where('file_id >= ?', (int) $position + 1) ->order('file_id ASC') ->limit(1));

     

    Casey said:

     

    Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.

    Yeah maybe at one point I'll go that route. For 200GB+ I'm only paying roughly $10/month so it's not that bad for me. The issue is that my members upload a lot. Maybe I'll use a Host Gator shared account for the "unlimited" storage? What do you think?


    This post was edited by socialenginestaff at October 24, 2017 4:58 AM EDT
    • 66 posts
    October 24, 2017 12:18 AM EDT

     

    Change line 33.... the limit is set to 5... change the 5 to whatever...try 100....

    // Prepare $service_id = $this->getParam('service_id'); $position = $this->getParam('position', 0); $progress = $this->getParam('progress', 0); $total = $this->getParam('total'); $limit = $this->getParam('limit', 5); $isComplete = false; $break = false;
    • 66 posts
    October 24, 2017 12:21 AM EDT

    Are your users multi-geographical? Or are they roughly in the same area/location?

    • 66 posts
    October 24, 2017 12:30 AM EDT

    oh...BTW.... this method is only for transferring ALL files between storage systems. If you want to move all files off local storage to AWS or to another 3rd party storage server or all files back to your local storage. It will not take 9 days in the future to make the changes. To add an answer to your question....S3 upload speeds are just slow for everyone. You need to look at S3 Transfer Acceleration. Not sure if it will fork with SE.

    • 66 posts
    October 24, 2017 12:32 AM EDT

    http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html

     

    Use this tool to check differences

    • Moderator
    • 6923 posts
    October 24, 2017 4:59 AM EDT

    @OTAw, I edited your post to add code tags instead of deleting the code per our posted terms. Please do use the code tags as explained in our posting guidelines for any code you post. Thank you! Stickied guidelines for posting are at the top of the forum sections. 

    Great thread guys! Thanks @Casey for your input too.

    • 275 posts
    June 6, 2020 2:41 AM EDT

    Awesome post guys.

     

    I am planning to implement AWS S3 as prescribed in this post.


    The plan is to make S3 the final and only storage place for the site and use Transfer Acceleration.

    Are there any updates on this strategy ? any new things to watch out for?

    Thanks


    This post was edited by playmusician at June 6, 2020 2:46 AM EDT
    • 17 posts
    December 18, 2020 6:15 AM EST

    I'm searching this thread, is there any detailed guide also on how to prepare the server on Amazon s3 side?  Since I'm also interested in getting more servers around the globe.. to make it work faster..

     

    Best regards, G

    • Moderator
    • 6923 posts
    December 18, 2020 7:12 AM EST

    Did you search our Learning Center knowledebase? https://www.socialengine.com/support/article/5243168/se-php-how-to-use-the-cloud-storage-feature that tutorial should help some.

  • dgw
    • 8 posts
    December 9, 2021 9:06 PM EST

    This link isn't working:  https://www.socialengine.com/support/article/5243168/se-php-how-to-use-the-cloud-storage-feature

    Is there a more recent link?

     

    I am trying to transfer files to S3 and I am getting errors, so I need to know exactly how to set up the S3 side.

    • Moderator
    • 6923 posts
    December 10, 2021 5:04 AM EST

    FYI, the Learning Center link in the menus above goes to the tutorials. The link you have is old. Here's the new one https://socialengine.atlassian.net/wiki/spaces/SU/pages/5243168/SE+PHP+-+How+to+use+the+Cloud+Storage+Feature . Just making you aware of where the new learning center is as well.

  • dgw
    • 8 posts
    December 10, 2021 8:35 AM EST

    Thanks, but I have followed these instructions exactly and I am getting these unhelpful errors in the job log when the transfer job runs:

     

     

    • File 1 could not be moved for reason: Unable to store file.
    • File 2 could not be moved for reason: Unable to store file.
    • File 3 could not be moved for reason: Unable to store file.
    • File 4 could not be moved for reason: Unable to store file.

    I was hoping for a bit more detail than "create an S3 account and get the security credentials and enter them into the storage service page"

    I want to get all of the files in the public folder onto S3 storage so I can upgrade to SE5 because our hosting provider says that the hundreds of thousands of files in public are causing the upgrades to fail.