The 64Base Dollar Question .(user upload file limits)

    • 275 posts
    February 7, 2019 12:37 PM EST

    So heres an interesting issue regarding users file size upload limit (video)

     

    Basically users can upload any video upload file size - even if the administrator has restricted the settings in php.ini

     

    Our Cpanel php.ini (7.2) setting is restricted to 15 M for  uploads by users

    post_max_size  = 15 M
    upload_max_filesize = 15 M

     

    We ran a series of tests to find out if this video upload was being adhered to.

    When uploading videos through the seao video plugin 

    we found we could upload nearly any size , the php.ini restriction we set really didnt matter.

    After not finding any .htaccess restricting this, we were very puzzled and 

    we decided to approach cpanel to see how this was possible.

     

    They did a battery of tests ( below) and found that the seao scripts are permitting or bypassing the php.ini settings for uploads.

     

    It boils down to the Base64 encoding.

     

    My questions are

     

    #1 should the seao video upload  scripts be bypassing the php.ini settings?

     

    #2 there are many upload forms on the site (audio,video,photos,documents and forms),  which leaves the admin of the site incapable of controlling the upload limits for user files

     

    #3 how can this be fixed?

     

    #4 how can the administrator of a site gain back control of how much we want the upload size limit to be?

     

    Thanks !!

     

     

     

    CPANELS Findings regarding increasing the upload limit

    ------------------------------CPanel Findings-------------------------------------------

    Thank you for your patience.

    I found that your scripts appear to be manually setting the ini files:

    ================================
    [11:06:38 fast root@11381393 /home/playmusician/public_html]cPs# grep 100M ./application/modules/Seaocore/Api/Gmail/googleapi.php ./application/modules/Sitebackup/controllers/AdminBackupsettingsController.php ./application/modules/Siteevent/controllers/AdminImporteventController.php ./application/modules/Sitevideointegration/controllers/AdminImportvideoController.php
    ./application/modules/Seaocore/Api/Gmail/googleapi.php: ini_set('upload_max_filesize', '100M');
    ./application/modules/Seaocore/Api/Gmail/googleapi.php: ini_set('post_max_size', '100M');
    ./application/modules/Sitebackup/controllers/AdminBackupsettingsController.php: ini_set('upload_max_filesize', '100M');
    ./application/modules/Sitebackup/controllers/AdminBackupsettingsController.php: ini_set('post_max_size', '100M');
    ./application/modules/Siteevent/controllers/AdminImporteventController.php: ini_set('upload_max_filesize', '100M');
    ./application/modules/Siteevent/controllers/AdminImporteventController.php: ini_set('post_max_size', '100M');
    ./application/modules/Sitevideointegration/controllers/AdminImportvideoController.php: ini_set('upload_max_filesize', '100M');
    ./application/modules/Sitevideointegration/controllers/AdminImportvideoController.php: ini_set('post_max_size', '100M');
    ================================

    I backed these files up to "/root/cptechs/11381393" and updated them to 15M:
    ================================
    [11:06:53 fast root@11381393 /home/playmusician/public_html]cPs# sed 's/100M/15M/' ./application/modules/Seaocore/Api/Gmail/googleapi.php ./application/modules/Sitebackup/controllers/AdminBackupsettingsController.php ./application/modules/Siteevent/controllers/AdminImporteventController.php ./application/modules/Sitevideointegration/controllers/AdminImportvideoController.php | grep 15M
    ini_set('upload_max_filesize', '15M');
    ini_set('post_max_size', '15M');
    ini_set('upload_max_filesize', '15M');
    ini_set('post_max_size', '15M');
    ini_set('upload_max_filesize', '15M');
    ini_set('post_max_size', '15M');
    ini_set('upload_max_filesize', '15M');
    ini_set('post_max_size', '15M');
    [11:07:31 fast root@11381393 /home/playmusician/public_html]cPs# sed -i 's/100M/15M/' ./application/modules/Seaocore/Api/Gmail/googleapi.php ./application/modules/Sitebackup/controllers/AdminBackupsettingsController.php ./application/modules/Siteevent/controllers/AdminImporteventController.php ./application/modules/Sitevideointegration/controllers/AdminImportvideoController.php
    ================================

    Unfortunately, the website is still able to upload the large video.

     

    This confirms that the issue is caused by the coding in your website software, and if the limit needs to be decreased, this must be resolved by the developer of the website. My thoughts are that perhaps the code that is increasing the limit is base64 (or otherwise) encoded, such that it is not readable.


    Kind Regards,

     

     


    This post was edited by playmusician at February 10, 2019 3:12 PM EST
    • Moderator
    • 6923 posts
    February 7, 2019 5:21 PM EST

    Wow this would be something that clients should be aware of and have to agree to. In case some are wanting to limit stuff. I limit everything on my site per member level. I suggest you make the devs you deal with aware of this thread in case others do it too. This is a good discussion to bring up. 

    • 275 posts
    February 7, 2019 6:09 PM EST

    yup , have let them know. Thanks

    perhaps, one solution could be

    -if there were "settings" made available to us for upload limits of each (audio,video,photos,documents or any upload forms) in the plugins ,

    which we could tweak or set as required, it would help.

     

    Cheers !~


    This post was edited by playmusician at February 10, 2019 3:56 PM EST
    • 275 posts
    February 9, 2019 3:34 PM EST

    All 3 seao plugins (audio/photo & video)

    bypass the php.ini post_max_size and  upload_max_filesize  settings in php.ini

     

    We have been able to upload files larger than 15 MB (set in our php.ini)

     

    This is a real serious issue.

    cause it bypasses the security of the site.


    This post was edited by playmusician at February 10, 2019 3:05 PM EST
    • Moderator
    • 6923 posts
    February 10, 2019 2:38 PM EST

    I discussed this with our CTO and we won't allow this sort of bypass. Our terms for the marketplace will be updated and if any clients report this to us we will disable any plugins that do this. Please let SEAO know of this thread so they are aware of our policy change regarding this topic as we won't be doing an announcement. If you have certified plugins that are doing this, please pm me the details and I'll disable them this week. 

    This sort of hidden change is not something we want. Any changes by third party plugins to server settings need to be checked with the client first. It cannot be something unknown and unapproved. 

    Our CTO: "It will be in our terms that third party products cannot modify the servers ini settings such as those mentioned. This is a flaw technically in PHP or some might call it a feature however servers can disable the override nature which in this case the server does not."

    I made bold that last part in case you wanted to get your host to do something to prohibit such changes in case a plugin does try it again.

  • gs
    • 857 posts
    February 10, 2019 2:53 PM EST

    I'm slightly confused (could be old age though) - are the settings simply being bypassed/ignored, or are the Plugins actually modifying the .ini file settings?

    • 275 posts
    February 10, 2019 2:55 PM EST

    Thanks Donna,

    have updated them of the same in a couple of tickets on their site.

     

    • 275 posts
    February 10, 2019 2:57 PM EST

    gs, they are not modifying the actual php.ini settings or file ,

     

    but just like an .htaccess file can override the settings by php directives,  if permitted

    its being overridden by code

     

    even though cpanel reduced it back manually, it still was getting overriden and they suspect it could additionally be the base64 encoder.

     

    Since uploads (audio,video,photos,documents and forms) are scattered all over the site in various pages ,

    this becomes a serious security issue for admins.

    we will have to fix this at some point

     

    Cpanel: My thoughts are that perhaps the code that is increasing the limit is base64 (or otherwise)encoded, such that it is not readable.


    This post was edited by playmusician at February 10, 2019 3:11 PM EST
    • 275 posts
    February 10, 2019 3:18 PM EST

    Another issue , we are trying to solve with them is - the music plugin is uploading files without the validation kicking in and no alert being given to the user who is uploading the files whether its a successful upload or not,  

    we always knew that there were many many "orphaned" files recorded in the database as well as lying on the server causing server "bloat" , due to not being deleted totally through the pages, but could not track or prove it till now ..until...

     

    we recently commissioned and designed a plugin to track uploaded & stored files on the server, which has become a real eye opener as we can troubleshoot every file uploaded which is stored. and we found when users try to upload files , he thinks its not uploaded 

    - as there were no alerts 

    - the validation was not working

    We are trying to fix that too. as you can tell we have tickets open) and hope to resolve this expeditely.

     

     With this new plugin we have designed,

    -we can also tell if any deletion on the site through a page works or not (like an admin deleting a member and all his respective(audio/video/photos) files. We are sifting to make sure that files are actually deleted and dont wind up "orphaned" taking up space on the server, which costs us money.

     

    Uploads are serious business and hopefully we can help make these plugins robust and efficient by helping troubleshoot.

     


    This post was edited by playmusician at February 10, 2019 3:41 PM EST
    • 275 posts
    February 10, 2019 4:16 PM EST

    "...I made bold that last part in case you wanted to get your host to do something to prohibit such changes in case a plugin does try it again."

    Donna,

    "servers can disable the override nature which in this case the server does not.""

    Unfortunately , we are stuck between a rock and a hard place regarding this

    If we disable exec, system (php functions)

    - FFMPEG will not work - which we need for encoding video (with video plugins)

    Its an issue we see everywhere , how to make FFMPEG work without enabling exec and system.

    And are looking into it.

     

    Thanks!


    This post was edited by playmusician at February 10, 2019 4:20 PM EST
    • 348 posts
    February 11, 2019 1:37 AM EST
    playmusician said:

    So heres an interesting issue regarding users file size upload limit (video)

     

    Basically users can upload any video upload file size - even if the administrator has restricted the settings in php.ini

     

    Our Cpanel php.ini (7.2) setting is restricted to 15 M for  uploads by users

    post_max_size  = 15 M
    upload_max_filesize = 15 M

     

    We ran a series of tests to find out if this video upload was being adhered to.

    When uploading videos through the seao video plugin 

    we found we could upload nearly any size , the php.ini restriction we set really didnt matter.

    After not finding any .htaccess restricting this, we were very puzzled and ............................ 

     

    playmusician

    Apologies for the delay in getting back to you regarding this concern here. Yes, you are correct here! We allow upload of large files, but we do not ignore the values set in php.ini we use a different upload process of uploading file in our plugins and i.e. we upload file in segments. 

    php.ini limits.....

    We upload content in chunks in our modules and not as a whole. The value set in php.ini file in this case validates the size of each chunk i.e. the size of a chunk can not be greater than that set in the php.ini so, we do not bypass this value.

    We have implemented this functionality as we had many clients facing an issue while trying to upload large size files (files of size greater than the one set in php.ini). The value set in php.ini file is bit less and users face issue while trying to upload large sized files.

    We have had a discussion with our team to find a workaround for this so that your concern also gets addressed. Once we are done with implementing the solution, we will post an update here for the same.


    CPanel findings.....

    - googleapi.php
    We will remove the restrictions and will do necessary corrections.

    - data import
    We increase the limit here as these files (import and backup files) are usually large in size. As we can not directly edit the .htaccess file, we have modified our code to accomodate the large files.

    Also, we would like to add that this ability is available only for backend functionalities.


    --> Regarding other concerns/questions, I will answer them here shortly.



    Thank You.


    This post was edited by SocialEngineAddOns at February 11, 2019 1:38 AM EST
    • Moderator
    • 6923 posts
    February 11, 2019 5:50 AM EST

    We can't allow something that overrides a client's settings. What this does is bypass the settings a client thinks they have nailed down, such as post_max_size  = 15 M , upload_max_filesize = 15 M . It is never a good idea to circumvent settings a client has determined is best for their site. Also, it's best to educate clients in regards to the settings, IE: "We have implemented this functionality as we had many clients facing an issue while trying to upload large size files (files of size greater than the one set in php.ini)"  Rather than force all clients to accept your determination of what is good for upload sizes, you should have educated those clients as to why the large files cannot be uploaded, to contact their host about raising limits and also the security concerns of raising such limits. 

    Thank you for working on a solution to remove this from all of your plugins and to allow the admin to choose whether they want larger files to be uploaded. Also, for educating clients as to why they should or shouldn't (pluses and minuses of) raise the limits and how they do that. 

    • Moderator
    • 6923 posts
    February 11, 2019 6:59 AM EST

    Also, this isn't good to do 

    ini_set("memory_limit", "1024M"); set_time_limit(0);
    • 348 posts
    February 11, 2019 7:56 AM EST
    Donna said:

    We can't allow something that overrides a client's settings. What this does is bypass the settings a client thinks they have nailed down, such as post_max_size  = 15 M , upload_max_filesize = 15 M . It is never a good idea to circumvent settings a client has determined is best for their site. Also, it's best to educate clients in regards to the settings, IE: ".......

    Hi Donna

    - We can't allow something that overrides a client's settings. What this does is bypass the settings a client thinks they have nailed down.... 

    Thanks for the insights here. We would like to add that, we are not following this approach globally on the website, this is being used in specific cases only. And these specific cases are like while uploading any video file, music file, any HD image, it could also be the case that multiple websites are being hosted on single server, in these cases applying php.ini check may not be of much use so, for such instances we follow this approach. 

    Another example of one such case where the value of the parameter 'upload_max_filesize' has been altered is, in SocialEngine functionality, for installing plugins we check the value set for 'upload_max_filesize' in .htaccess file in install, this limit has been increased so as to accommodate the large sized plugin files. Now, this has been implemented in this case as usually plugin file size is bigger in size and default value set in php.ini will not let the plugin installed on the site. 


    - Rather than force all clients to accept your determination of what is good for upload sizes, you should have....

    We address such client concerns by first explaining them that how the SocialEngine core architecture is and how it functions then only we proceed with helping them out to achieve their requirements. In this particular case, clients usually do not prefer making the change directly in the php.ini values and restrain themselves from contacting their hosting provider. 
    This request has been made by a large no of clients and because of this only we implemented it in this way in our plugins. 

    To address the concerns shared by playmusician and you, we are working on implementing some functionality using which Site Admin may decide that whether they want to apply the checks for file uploads as per the limits set in php.ini or not. 

    This solution will address the user-end concern, users will not be able to upload files of size greater than the one specified in php.ini (if set as such by the site admin), but there are few instances at the admin-end like, import file, backup files etc. these files are usually quite big in size so implementing check for limit set in php.ini will not work. 

    Thank You. 


    This post was edited by SocialEngineAddOns at February 11, 2019 7:59 AM EST
    • Moderator
    • 6923 posts
    February 11, 2019 8:21 AM EST

    Don't overlook the memory limits and the max_execution you also override. We won't be allowing these or the other changes via the plugin without transparency and admin approval. That approval must be after admin is informed of the security and server concerns in allowing those large values. Clients upping the mem to that huge amount risk server malfunctions/down time or needing to get a larger server at a greater cost. It would be far better for the plugins to be optimized rather than requiring clients to risk security issues or server issues / higher cost. 

    • Moderator
    • 6923 posts
    February 11, 2019 8:23 AM EST

    Oh and the errors that are also disabled from displaying. Basically, anything you are changing with the ini_set like this:

    ini_set('memory_limit', '2048M'); set_time_limit(0); ini_set('upload_max_filesize', '100M'); ini_set('post_max_size', '100M'); ini_set('max_input_time', 600); ini_set('max_execution_time', 600); ini_set('display_errors', FALSE); error_reporting(0); ini_set("memory_limit", "1024M"); set_time_limit(0);
    • Moderator
    • 6923 posts
    February 11, 2019 8:24 AM EST

    We aren't trying to argue or make things difficult. We just want transparency and a better experience for clients.

  • gs
    • 857 posts
    February 11, 2019 9:12 AM EST

    as always, I prefer having options, which it seems SEAO will be implementing.  ADMIN should be allowed to determine what they prefer based on info provided (i.e. transparency), including if a different VPS/server is warranted.  I don't need to be told what to do, but do expect and appreciate transparency and info

     

    Thanks @playmusician for posting this, @donna for useful input, and SEAO for offering a flexible solution so quickly.

    • 275 posts
    February 11, 2019 12:47 PM EST

    Thanks @Seao, @Donna for addressing this so efficiently and making this transparent.

     

    Regarding..

    ""... it could also be the case that multiple websites are being hosted on single server, in these cases applying php.ini check may not be of much use so, for such instances we follow this approach. .."

    We do host and manage our own server and have multiple websites.

    As admins, we always have the .htaccess option for >>>each site << to delegate what max limit etc should be. 

     

    This has been great information and hopefully lets keep up the transparency to make the experience and products better for our users and the community as a whole.

     

    As We've suggested before an online technical specs document

    >> specifically  for admins ( with back end technical info) would solve a lot of heartache regarding these issues, we have many other such issues (example FFMPEG which is required for your video plugin, will not work without disabling exec and system in php.ini - which is  a serious security issue) 

    which just by opening tickets - we find do  not get addressed and answered properly . Hope we can fix that for the future.

    Thank You:)

     


    This post was edited by playmusician at February 11, 2019 1:03 PM EST
    • 275 posts
    February 11, 2019 3:21 PM EST

    Also, we would like some clarity on this statement...

    "We upload content in chunks in our modules and not as a whole. The value set in php.ini file in this case validates the size of each chunk i.e. the size of a chunk can not be greater than that set in the php.ini so, we do not bypass this value."

    There are 2 separate settings

    post_max_size  
    upload_max_filesize 

     

    My 2 cents..

    post_max_size is the maximum size for all POST body data. It doesn't matter if you're POSTing JSON or your DVD collection, this is all POST body data.

    if you are uploading multiple files, the total file size has to fit within this limit.

    upload_max_filesize is a maximum size only for files that are POSTED. Other types of POST body data are not subject to this limit.

     

    When you say..

    The value set in php.ini file in this case validates the size of each chunk i.e. the size of a chunk can not be greater than that set in the php.ini

     

    >>> Im assuming your referring to  upload_max_filesize.

     

    But in this case, you are bypassing the post_max_size  

    since it also has been set as 15 M and bypassed it.

     

    Thanks

    • 275 posts
    February 11, 2019 7:18 PM EST

    also regarding

    "As we can not directly edit the .htaccess file, have modified our code to accomodate the large files."

     

    There are 2 issues here

    #1 the hosting company might not allow .htaccess override with php directives

    - in that case inform the admin to request it.

    - if that request fails , let the admin know that the plugin will not work correctly

     

    #2 assuming .htaccess override is permitted by the hosting company

    - let the admin know that these changes would occur.

    - provide them an "interface" where they can input the desired settings

     

    Another example where we see this issue is

    Seao's video plugin (like others) requires FFMPEG

    FFMPEG only works if the exec and system functions are disabled in PHP.ini

    - this being a security risk, would require the admin to request teh hosting company to disable these functions

    Again, you need to provide information and documentation what to do and how to do  - in order to make the plugin work.

    - and its effect of doing so.

    We need to share information transparently so we can get to the meat and bones of software security and risks. 

     

    As a wanna be social engine hosting artist platform, We would never go live and accept a single dollar from a user 

    - unless these risks were addressed in detail.

     


    Please let's not gloss over important issues, for the future..

    an informed client is a happy client:))


    Hope this helps

    Thanks

     

     

     


    This post was edited by playmusician at February 11, 2019 7:44 PM EST
    • 348 posts
    February 12, 2019 7:09 AM EST

    Hi playmusician & Donna,

    We have had a discussion regarding the ways to implement transparency as per the above exchanged messages. What we have thought of implementing is:

    1) We will provide a setting in the admin panel using which Site Admin may choose between 2 options: Applying the checks for file uploads as per the limits set in php.ini or Not applying the checks. 

    2) We will provide an FAQ in each of the module which make use of the above setting elaborating the pros and cons for choosing the given setting. A proper explanation will be added which will contribute to the transparency in this case.

    Please let us know in case there is any another suggestion to fulfil the 'Transparency' criteria. 

     

    playmusician 
    Regarding FFMPEG functioning: exe function is required to run FFMPEG through php script. We have alternatives in the market, but those alternatives do not work directly with the php script. We will be discussing this with team for finding out an alternate solution here. Once we have the alternate solution, we will inform your personally for it.

    Thank You.


    This post was edited by SocialEngineAddOns at February 12, 2019 7:10 AM EST
    • Moderator
    • 6923 posts
    February 12, 2019 8:03 AM EST

    @SEAO, your transparency suggestions should work fine. 

    As for FFMPEG even the SE Video plugin requires it. Just so everyone knows.

    • 275 posts
    February 12, 2019 10:33 AM EST

    That sounds good SEAO,

    gives the admin a solid choice with transparency

    The FAQ will help too.

     

    Looking forward to the FFMPEG solution.

     

    Glad we could be of help too.

    Thanks SEAO & Donna .

     


    This post was edited by playmusician at February 12, 2019 11:56 AM EST
    • 348 posts
    February 13, 2019 5:38 AM EST

    Thanks Donna for the confirmation. We will notify here once we have implemented the required work. 

    playmusician
    For FFMPEG Solution: we will contact you personally for this once we have found a workaround.

    Thank You.