Fixing a file streaming bug in a Rails app
A Rails app I work on accepts file uploads from public-facing websites. Uploaded files may contain sensitive information. We offer this so people don’t have to email their sensitive documents to our customers.
When a file is uploaded, our app generates a new asymmetric encryption key pair and uses the public key to encrypt the uploaded file. The encrypted file is then stored, and the private key is stored separately.
The app then makes the original file available to our customers to download. When a download request is initiated, the app retrieves the encrypted file and the private key, decrypts the file, streams it to the client, and then deletes the decrypted file, so that no sensitive info is left lying around on our server.
Here’s a simplified version of the code I’m talking about, showing a Rails controller action (
download) that streams a file, and a private method the action depends on (
with_decrypted_file) that decrypts a file, yields it to a block, and then deletes it.
def download form_post = FormPost.for(current_user).find(params[:id]) with_decrypted_file(form_post) do |file| # Pathname send_file_headers!( disposition: 'attachment', filename: file.basename) response.delete_header('Content-Length') self.response_body = Enumerator.new do |chunks| file.open('r') do |f| chunks << f.read(8192) while !f.eof? end end end end private def with_decrypted_file(form_post) encrypted_file = form_post.fetch_decrypted_file decrypted_file = nil begin decrypted_file = EncryptionHandler.decrypt( encrypted_file, form_post.private_key) fail if !decrypted_file.exist? yield(decrypted_file) ensure encrypted_file.delete if encrypted_file&.exist? decrypted_file.delete if decrypted_file&.exist? end end
Here is the same code as above, simplified even further to make it easier to see the meat of what’s going on.
def download with_decrypted_file do |file| self.response_body = Enumerator.new do |chunks| file.open('r') do |f| chunks << f.read(8192) while !f.eof? end end end end private def with_decrypted_file decrypted_file = fetch_and_decrypt_file yield(decrypted_file) ensure decrypted_file.delete end
I didn’t have much experience about Rails response streaming when I wrote this code, but it seemed fairly straightforward and worked fine. Encrypted files were fetched, decrypted, streamed, and deleted. Everyone was happy.
Until we started seeing errors, that is. The errors were only happening in some cases, and even then they were sometimes not reproducible. We would get a user report that a download was failing, but we’d try it ourselves and it would work. We’d have the user try it again and it would work. 🤷♂️
Stranger still, there were two different errors that were occurring, but only one would occur at a time, the choice seemingly random:
Errno::ENOENT: No such file or directory @ rb_sysopen
IOError: closed stream
At first this didn’t make any sense. There’s an explicit
fail if the decrypted file doesn’t exist (a sanity check to make sure decryption didn’t fail silently), but it wasn’t being tripped. The file definitely existed, and it wasn’t being deleted (cleaned up) until after the
yield block had finished.
Right? Wrong. A look under the hood revealed that Rails streaming happens in a separate thread. So we had a race condition, which explained why this issue only happens sometimes and is difficult to reproduce. Turns out it was happening less often on smaller files (maybe under 250K), when reading the file in one thread could finish before the file was deleted in another thread.
I couldn’t find much in the way of how to solve this, such as a
flush method on the response stream, but I found my way to the
Response#await_sent method, which is exactly what was needed. (There’s also
Response#await_commit, but that wasn’t enough for this case.)
response.await_sent as the last line of the
with_decrypted_file block fixed the issue by eliminating the race condition. The decrypted file is now guaranteed to finish being streamed before it gets deleted.
Here’s the updated
download action method:
def download form_post = FormPost.for(current_user).find(params[:id]) with_decrypted_file(form_post) do |file| # Pathname send_file_headers!( disposition: 'attachment', filename: file.basename) response.delete_header('Content-Length') self.response_body = Enumerator.new do |chunks| file.open('r') do |f| chunks << f.read(8192) while !f.eof? end end # wait to exit the block until the file has been # streamed so it doesn't get deleted beforehand response.await_sent end end
I hope this post saves someone else some time if they are greeted with intermittent
Errno::ENOENT errors while using a
response_body Enumerator to stream files in Rails.