Community

Forums » SEPHP Help and Tips

Amazon S3 as default storage?

    • 14 posts
    October 11, 2017 2:43 PM EDT

    Hey all,

     

    I spent the better half of 9 days transferring 200+GB worth of files to an Amazon S3 bucket via the transfer tool under "Manage Storage Services". All of the files transferred fine like expected, but now when selecting S3 as my default storage, files take an extremely long time to upload. When I revert back to my server as the storage, everything uploads quickly.

     

    Using S3 as my default storage, a 1MB file will take a minute or so to upload, a 3MB maybe takes 3-5minutes, a 20MB file takes 5+ minutes to upload and so on. I love the idea of having this external storage for my site because my members upload a lot of content, but this slow upload using S3 is not a viable option. 

     

    So, is anybody else using S3 as their default storage? How is it working for you when uploading files? Slow? Fast? I'd love to get some input on this.

    • 920 posts
    October 12, 2017 4:23 AM EDT

    I think it depends on the region you selected. I recall seeing something about this for someone else. https://forums.aws.amazon.com/thread.jspa?threadID=116513 try a traceroute and there's one mentioned by Amazon in that thread. See if it shows any issues. We use Amazon here and at our demo and hadn't noticed an issue.  

    • 14 posts
    October 12, 2017 9:09 AM EDT

    Hi Donna,

     

    Thanks for the response.

     

    I am located in Dallas, my server is located in Chicago and my bucket is located in N.California. Maybe moving my files over to a bucket closer to my server (Ohio?) would improve a tiny bit? I'll review that thread in the link you provided.

     

    It should also be noted that I'm using SE 4.8.12 on PHP 5.6 (apache 3). Not sure if these outdated versions are adding to the problem but I won't be positive until Saturday when I upgrade to PHP 7.x (apache 4) and the latest version of social engine/plugins. I'll follow up to this thread once everything is upgraded.

     

    But in the mean time, you said the social engine demo uses S3 as the default storage location? Because browsing around the demo and viewing source on profile pictures, uploaded music etc. shows the files to be uploaded locally. 

     

    Example:

    Profile photo: https://demo.socialengine.com/profile/gary  --> https://demo.socialengine.com/public/user/1a/01/f79e96a5e722ebe1c115e678c8fc1528.jpg

    Music: https://demo.socialengine.com/music/2/travelin-039-man  --> https://demo.socialengine.com/public/music_song/35/01/b59af033c23a2ba8bb3d330eda39677e.mp3

    • 920 posts
    October 12, 2017 9:20 AM EDT

    You should upgrade as there were a lot of improvements since that version. I'll double check the demo as I thought it was using S3 like the rest of our sites. 

    • 920 posts
    October 12, 2017 9:23 AM EDT

    Ok yeah it uses Amazon but since it gets reset, the main files in there for the demo content are stored elsewhere due to the reset each day. Newly uploaded stuff for people testing should be on Amazon if you test by signing up and loading a photo or something. This site also uses Amazon, just double checked.

    • 14 posts
    October 14, 2017 8:08 PM EDT

    Thanks for the feedback, Donna. I've upgraded to the latest version of SE4 as well as PHP. I will update you all how S3 upload speeds are in a few days, now that I've upgraded.

    • 29 posts
    October 23, 2017 11:42 PM EDT

    You can actually go edit a core SE file to make the transfer go a lot faster in the future. For some reason SE will only transfer a few files at one time but you can edit the limit for the transfer to make it send more at one time.

     

    "/application/modules/Storage/Plugin/Job/Transfer.php"

    If you look in there, you can change it for future transfers. Just do not use too large of a number or the transfer may fail or timeout.

     

    Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.


    This post was edited by Casey at October 23, 2017 11:44 PM EDT
    • 14 posts
    October 24, 2017 12:09 AM EDT

    Casey said:

     

    "/application/modules/Storage/Plugin/Job/Transfer.php"

    If you look in there, you can change it for future transfers.

     

     Are you referring to this instance?:

    $file = $filesTable->fetchRow($filesTable->select() ->where('service_id != ?', $service_id) ->where('file_id >= ?', (int) $position + 1) ->order('file_id ASC') ->limit(1));

     

    Casey said:

     

    Also, if you can afford another server.... make your own storage server and forget AWS's high price. The Virtual File System option is the way to go and then you can do LB and have all your Webservers pull member uploaded files from there.

    Yeah maybe at one point I'll go that route. For 200GB+ I'm only paying roughly $10/month so it's not that bad for me. The issue is that my members upload a lot. Maybe I'll use a Host Gator shared account for the "unlimited" storage? What do you think?


    This post was edited by Donna at October 24, 2017 4:58 AM EDT
    • 29 posts
    October 24, 2017 12:18 AM EDT

     

    Change line 33.... the limit is set to 5... change the 5 to whatever...try 100....

    // Prepare $service_id = $this->getParam('service_id'); $position = $this->getParam('position', 0); $progress = $this->getParam('progress', 0); $total = $this->getParam('total'); $limit = $this->getParam('limit', 5); $isComplete = false; $break = false;
    • 29 posts
    October 24, 2017 12:21 AM EDT

    Are your users multi-geographical? Or are they roughly in the same area/location?

    • 29 posts
    October 24, 2017 12:30 AM EDT

    oh...BTW.... this method is only for transferring ALL files between storage systems. If you want to move all files off local storage to AWS or to another 3rd party storage server or all files back to your local storage. It will not take 9 days in the future to make the changes. To add an answer to your question....S3 upload speeds are just slow for everyone. You need to look at S3 Transfer Acceleration. Not sure if it will fork with SE.

    • 29 posts
    October 24, 2017 12:32 AM EDT
    • 920 posts
    October 24, 2017 4:59 AM EDT

    @OTAw, I edited your post to add code tags instead of deleting the code per our posted terms. Please do use the code tags as explained in our posting guidelines for any code you post. Thank you! Stickied guidelines for posting are at the top of the forum sections. 

    Great thread guys! Thanks @Casey for your input too.