Merge pull request #3993 from MrZYX/sidekiq

Sidekiq (+ some cleanup)

.foreman

@@ -1,2 +1,2 @@
 port: 3000
-formation: web=1,worker=0
+formation: web=1,sidekiq=0

Changelog.md

@@ -1,5 +1,102 @@
 # Head
 
+## Refactor
+
+### Replaced Resque with Sidekiq - Migration guide - [#3993](https://github.com/diaspora/diaspora/pull/3993)
+
+We replaced our queue system with Sidekiq. You might know that Resque needs Redis.
+Sidekiq does too, so don't remove it, it's still required. Sidekiq uses a threaded
+model so you'll need far less processes than with Resque to do the same amount
+of work.
+
+To update do the following:
+
+1. Before updating (even before the `git pull`!) stop your application
+   server (Unicorn by default, started through Foreman).
+2. In case you did already run `git pull` checkout v0.0.3.2:
+   
+   ```
+   git fetch origin
+   git checkout v0.0.3.2
+   bundle
+   ```
+   
+3. Start Resque web (you'll need temporary access to port 5678, check
+   your Firewall if needed!):
+   
+   ```
+   bundle exec resque-web
+   ```
+   
+   In case you need it you can adjust the port with the `-p` flag.
+4. One last time, start a Resque worker:
+   
+   ```
+   RAILS_ENV=production QUEUE=* bundle exec rake resque:work
+   ```
+
+   Visit Resque web via http://your_host:5678, wait until all queues but the
+   failed one are empty (show 0 jobs).
+5. Kill the Resque worker by hitting Ctrl+C. Kill Resque web with:
+   
+   ```
+   bundle exec resque-web -k
+   ```
+
+   Don't forget to close the port on the Firewall again, if you had to open it.
+6. In case you needed to do step 2., run:
+   
+   ```
+   git checkout master
+   bundle
+   ```
+
+7. Proceed with the update as normal (migrate database, precompile assets).
+8. Before starting Diaspora again ensure that you reviewed the new
+   `environment.sidekiq` section in `config/diaspora.yml.example` and,
+   if wanted, transfered it to your `config/diaspora.yml` and made any
+   needed changes. In particular increase the `environment.sidekiq.concurrency`
+   setting on any medium sized pod. If you do change that value, edit
+   your `config/database.yml` and add a matching `pool: n` to your database
+   configuration. n should be equal or higher than the amount of
+   threads per Sidekiq worker. This sets how many concurrent
+   connections to the database ActiveRecord allows.
+
+
+If you aren't using `script/server` but for example passenger, you no
+longer need to start a Resque worker, but a Sidekiq worker now. The
+command for that is:
+
+```
+bundle exec sidekiq
+```
+
+
+#### Heroku
+
+The only gotcha for Heroku single gear setups is that the setting name
+to spawn a background worker from the unicorn process changed. Run
+
+```
+heroku config:remove SERVER_EMBED_RESQUE_WORKER
+heroku config:set SERVER_EMBED_SIDEKIQ_WORKER=true
+```
+
+We're automatically adjusting the ActiveRecord connection pool size for you.
+
+Larger Heroku setups should have enough expertise to figure out what to do
+by them self.
+
+### Other
+
+* Cleaned up requires of our own libraries [#3993](https://github.com/diaspora/diaspora/pull/3993)
+* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)
+* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)
+* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)
+* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)
+* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)
+* Delete unnecessary javascript views. [#4059](https://github.com/diaspora/diaspora/pull/4059)
+
 ## Bug fixes
 
 * Fix cancel button on getting_started confirmation box [#4073](https://github.com/diaspora/diaspora/issues/4073)
@@ -17,15 +114,6 @@
 * Fix off-center close button image on intro popovers [#3841](https://github.com/diaspora/diaspora/pull/3841)
 * Remove unnecessary dotted CSS borders. [#2940](https://github.com/diaspora/diaspora/issues/2940)
 
-## Refactor
-
-* Delete unnecessary javascript views. [#4059] (https://github.com/diaspora/diaspora/pull/4059)
-* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)
-* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)
-* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)
-* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)
-* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)
-
 ## Features
 
 * Deleting a post that was shared to Facebook now deletes it from Facebook too [#3980]( https://github.com/diaspora/diaspora/pull/3980)

Gemfile

@@ -17,8 +17,9 @@ gem 'devise', '2.1.3'
 
 # Background processing
 
-gem 'resque',         '1.23.0'
+gem 'sidekiq', '2.7.5'
-gem 'resque-timeout', '1.0.0'
+gem 'sinatra', '1.3.3'
+gem 'slim', '1.3.6'
 
 # Configuration
 

Gemfile.lock

@@ -62,6 +62,9 @@ GEM
     carrierwave (0.8.0)
       activemodel (>= 3.2.0)
       activesupport (>= 3.2.0)
+    celluloid (0.12.4)
+      facter (>= 1.6.12)
+      timers (>= 1.0.0)
     childprocess (0.3.9)
       ffi (~> 1.0, >= 1.0.11)
     chunky_png (1.2.7)
@@ -81,6 +84,7 @@ GEM
     compass-rails (1.0.3)
       compass (>= 0.12.2, < 0.14)
     configurate (0.0.2)
+    connection_pool (1.0.0)
     crack (0.3.2)
     cucumber (1.2.3)
       builder (>= 2.1.2)
@@ -103,6 +107,7 @@ GEM
     excon (0.20.1)
     execjs (1.4.0)
       multi_json (~> 1.0)
+    facter (1.6.17)
     factory_girl (4.2.0)
       activesupport (>= 3.0.0)
     factory_girl_rails (4.2.1)
@@ -320,13 +325,6 @@ GEM
     redis-namespace (1.2.1)
       redis (~> 3.0.0)
     remotipart (1.0.5)
-    resque (1.23.0)
-      multi_json (~> 1.0)
-      redis-namespace (~> 1.0)
-      sinatra (>= 0.9.2)
-      vegas (~> 0.1.2)
-    resque-timeout (1.0.0)
-      resque (~> 1.0)
     rmagick (2.13.2)
     roxml (3.1.6)
       activesupport (>= 2.3.0)
@@ -362,11 +360,20 @@ GEM
       multi_json (~> 1.0)
       rubyzip
       websocket (~> 1.0.4)
+    sidekiq (2.7.5)
+      celluloid (~> 0.12.0)
+      connection_pool (~> 1.0)
+      multi_json (~> 1)
+      redis (~> 3)
+      redis-namespace
     simple_oauth (0.2.0)
     sinatra (1.3.3)
       rack (~> 1.3, >= 1.3.6)
       rack-protection (~> 1.2)
       tilt (~> 1.3, >= 1.3.3)
+    slim (1.3.6)
+      temple (~> 0.5.5)
+      tilt (~> 1.3.3)
     slop (3.4.4)
     spork (1.0.0rc3)
     sprockets (2.2.2)
@@ -375,10 +382,12 @@ GEM
       rack (~> 1.0)
       tilt (~> 1.1, != 1.3.0)
     subexec (0.2.2)
+    temple (0.5.5)
     terminal-table (1.4.5)
     thor (0.17.0)
     tilt (1.3.6)
     timecop (0.6.1)
+    timers (1.1.0)
     treetop (1.4.12)
       polyglot
       polyglot (>= 0.3.1)
@@ -396,8 +405,6 @@ GEM
       kgio (~> 2.6)
       rack
       raindrops (~> 0.7)
-    vegas (0.1.11)
-      rack (>= 1.0.0)
     warden (1.2.1)
       rack (>= 1.0)
     webmock (1.8.11)
@@ -471,8 +478,6 @@ DEPENDENCIES
   rb-inotify (= 0.9.0)
   redcarpet (= 2.2.2)
   remotipart (= 1.0.5)
-  resque (= 1.23.0)
-  resque-timeout (= 1.0.0)
   rmagick (= 2.13.2)
   roxml (= 3.1.6)
   rspec-instafail (= 0.2.4)
@@ -480,6 +485,9 @@ DEPENDENCIES
   ruby-oembed (= 0.8.8)
   sass-rails (= 3.2.6)
   selenium-webdriver (= 2.31.0)
+  sidekiq (= 2.7.5)
+  sinatra (= 1.3.3)
+  slim (= 1.3.6)
   spork (= 1.0.0rc3)
   timecop (= 0.6.1)
   twitter (= 4.6.2)

Procfile

@@ -1,2 +1,2 @@
 web: bundle exec unicorn_rails -c config/unicorn.rb -p $PORT
-worker: env QUEUE=* bundle exec rake resque:work
+sidekiq: bundle exec sidekiq

Rakefile

@@ -7,7 +7,6 @@
 
 require File.expand_path('../config/application', __FILE__)
 require 'rake'
-require 'resque/tasks'
 
 # for rake 0.9.0
 module Diaspora

app/controllers/admins_controller.rb

@@ -1,5 +1,3 @@
-require Rails.root.join('lib', 'statistics')
-
 class AdminsController < ApplicationController
   before_filter :authenticate_user!
   before_filter :redirect_unless_admin

app/controllers/invitations_controller.rb

@@ -2,8 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'email_inviter')
-
 class InvitationsController < ApplicationController
 
   before_filter :authenticate_user!, :only => [:new, :create]

app/controllers/likes_controller.rb

@@ -2,8 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join("app", "presenters", "post_presenter")
-
 class LikesController < ApplicationController
   include ApplicationHelper
   before_filter :authenticate_user!

app/controllers/people_controller.rb

@@ -2,8 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join("lib", 'stream', "person")
-
 class PeopleController < ApplicationController
   before_filter :authenticate_user!, :except => [:show, :last_post]
   before_filter :redirect_if_tag_search, :only => [:index]

app/controllers/posts_controller.rb

@@ -2,8 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join("app", "presenters", "post_presenter")
-
 class PostsController < ApplicationController
   include PostsHelper
 

app/controllers/publics_controller.rb

@@ -2,11 +2,7 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'stream', 'public')
 class PublicsController < ApplicationController
-  require Rails.root.join('lib', 'diaspora', 'parser')
-  require Rails.root.join('lib', 'postzord', 'receiver', 'public')
-  require Rails.root.join('lib', 'postzord', 'receiver', 'private')
   include Diaspora::Parser
 
   skip_before_filter :set_header_data
@@ -53,7 +49,7 @@ class PublicsController < ApplicationController
 
   def receive_public
     FEDERATION_LOGGER.info("recieved a public message")
-    Resque.enqueue(Jobs::ReceiveUnencryptedSalmon, CGI::unescape(params[:xml]))
+    Workers::ReceiveUnencryptedSalmon.perform_async(CGI::unescape(params[:xml]))
     render :nothing => true, :status => :ok
   end
 
@@ -69,7 +65,7 @@ class PublicsController < ApplicationController
     @user = person.owner
 
     FEDERATION_LOGGER.info("recieved a private message for user:#{@user.id}")
-    Resque.enqueue(Jobs::ReceiveEncryptedSalmon, @user.id, CGI::unescape(params[:xml]))
+    Workers::ReceiveEncryptedSalmon.perform_async(@user.id, CGI::unescape(params[:xml]))
 
     render :nothing => true, :status => 202
   end

app/controllers/services_controller.rb

@@ -36,7 +36,7 @@ class ServicesController < ApplicationController
       fetch_photo = current_user.profile[:image_url].blank?
 
       current_user.update_profile(current_user.profile.from_omniauth_hash(user))
-      Resque.enqueue(Jobs::FetchProfilePhoto, current_user.id, service.id, user["image"]) if fetch_photo
+      Workers::FetchProfilePhoto.perform_async(current_user.id, service.id, user["image"]) if fetch_photo
 
       flash[:notice] = I18n.t 'services.create.success'
     else

app/controllers/streams_controller.rb

@@ -2,15 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join("lib", "stream", "aspect")
-require Rails.root.join("lib", "stream", "multi")
-require Rails.root.join("lib", "stream", "comments")
-require Rails.root.join("lib", "stream", "likes")
-require Rails.root.join("lib", "stream", "mention")
-require Rails.root.join("lib", "stream", "followed_tag")
-require Rails.root.join("lib", "stream", "activity")
-
-
 class StreamsController < ApplicationController
   before_filter :authenticate_user!
   before_filter :save_selected_aspects, :only => :aspects

app/controllers/tags_controller.rb

@@ -1,8 +1,6 @@
 #   Copyright (c) 2010-2011, Diaspora Inc.  This file is
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
-require Rails.root.join('app', 'models', 'acts_as_taggable_on', 'tag')
-require Rails.root.join('lib', 'stream', 'tag')
 
 class TagsController < ApplicationController
   skip_before_filter :set_grammatical_gender

app/controllers/users_controller.rb

@@ -3,9 +3,6 @@
 #   the COPYRIGHT file.
 
 class UsersController < ApplicationController
-  require Rails.root.join('lib', 'diaspora', 'exporter')
-  require Rails.root.join('lib', 'collect_user_photos')
-
   before_filter :authenticate_user!, :except => [:new, :create, :public, :user_photo]
 
   respond_to :html

app/helpers/markdownify_helper.rb

@@ -2,9 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'diaspora', 'markdownify')
-require 'redcarpet/render_strip'
-
 module MarkdownifyHelper
   def markdownify(target, render_options={})
 

app/models/account_deletion.rb

@@ -26,7 +26,7 @@ class AccountDeletion < ActiveRecord::Base
   end
 
   def queue_delete_account
-    Resque.enqueue(Jobs::DeleteAccount, self.id)
+    Workers::DeleteAccount.perform_async(self.id)
   end
 
   def perform!

app/models/jobs/base.rb

@@ -1,25 +0,0 @@
-#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
-#   licensed under the Affero General Public License version 3 or later.  See
-#   the COPYRIGHT file.
-
-module Jobs
-  class Base
-    Dir[Rails.root.join('app', 'models', 'jobs', 'mail', '*.rb')].each {|file| require file }
-    
-    #TODO these should be subclassed real exceptions
-    DUMB_ERROR_MESSAGES = [
-      "Contact required unless request",
-      "Relayable object, but no parent object found" ]
-
-    def self.suppress_annoying_errors(&block)
-      begin
-        yield
-      rescue => e
-        Rails.logger.info("error in job: #{e.message}")
-        unless DUMB_ERROR_MESSAGES.include?(e.message) 
-          raise e
-        end
-      end
-    end
-  end
-end

app/models/jobs/gather_o_embed_data.rb

@@ -1,16 +0,0 @@
-#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
-#   licensed under the Affero General Public License version 3 or later.  See
-#   the COPYRIGHT file.
-#
-
-module Jobs
-  class GatherOEmbedData < Base
-    @queue = :http_service
-
-    def self.perform(post_id, url)
-      post = Post.find(post_id)
-      post.o_embed_cache = OEmbedCache.find_or_create_by_url(url)
-      post.save
-    end
-  end
-end

app/models/jobs/receive.rb

@@ -1,17 +0,0 @@
-#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
-#   licensed under the Affero General Public License version 3 or later.  See
-#   the COPYRIGHT file.
-
-
-module Jobs
-  class Receive < Base
-
-    @queue = :receive
-    def self.perform(user_id, xml, salmon_author_id)
-      user = User.find(user_id)
-      salmon_author = Person.find(salmon_author_id)
-      zord = Postzord::Receiver::Private.new(user, :person => salmon_author)
-      zord.parse_and_receive(xml)
-    end
-  end
-end

app/models/jobs/receive_unencrypted_salmon.rb

@@ -1,21 +0,0 @@
-#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
-#   licensed under the Affero General Public License version 3 or later.  See
-#   the COPYRIGHT file.
-
-require Rails.root.join('lib', 'postzord', 'receiver', 'public')
-
-module Jobs
-  class ReceiveUnencryptedSalmon < Base
-    @queue = :receive
-
-    def self.perform(xml)
-      begin
-        receiver = Postzord::Receiver::Public.new(xml)
-        receiver.perform!
-      rescue => e
-        FEDERATION_LOGGER.info(e.message)
-        raise e
-      end
-    end
-  end
-end

app/models/notifications/also_commented.rb

@@ -1,6 +1,6 @@
 class Notifications::AlsoCommented < Notification
   def mail_job
-    Jobs::Mail::AlsoCommented
+    Workers::Mail::AlsoCommented
   end
   
   def popup_translation_key

app/models/notifications/comment_on_post.rb

@@ -1,6 +1,6 @@
 class Notifications::CommentOnPost < Notification
   def mail_job
-    Jobs::Mail::CommentOnPost
+    Workers::Mail::CommentOnPost
   end
 
   def popup_translation_key

app/models/notifications/liked.rb

@@ -1,6 +1,6 @@
 class Notifications::Liked < Notification
   def mail_job
-    Jobs::Mail::Liked
+    Workers::Mail::Liked
   end
   
   def popup_translation_key

app/models/notifications/mentioned.rb

@@ -1,6 +1,6 @@
 class Notifications::Mentioned < Notification
   def mail_job
-    Jobs::Mail::Mentioned
+    Workers::Mail::Mentioned
   end
   
   def popup_translation_key

app/models/notifications/private_message.rb

@@ -1,6 +1,6 @@
 class Notifications::PrivateMessage < Notification
   def mail_job
-    Jobs::Mail::PrivateMessage
+    Workers::Mail::PrivateMessage
   end
   def popup_translation_key
     'notifications.private_message'

app/models/notifications/request_accepted.rb

@@ -1,6 +1,6 @@
 class Notifications::RequestAccepted < Notification
   def mail_job
-    Jobs::Mail::RequestAcceptance
+    Workers::Mail::RequestAcceptance
   end
   def popup_translation_key
     'notifications.request_accepted'

app/models/notifications/reshared.rb

@@ -1,7 +1,6 @@
 class Notifications::Reshared < Notification
   def mail_job
-    Jobs::Mail::Reshared
+    Workers::Mail::Reshared
-    #Jobs::Mail::Liked
   end
 
   def popup_translation_key

app/models/notifications/started_sharing.rb

@@ -1,6 +1,6 @@
 class Notifications::StartedSharing < Notification
   def mail_job
-    Jobs::Mail::StartedSharing
+    Workers::Mail::StartedSharing
   end
 
   def popup_translation_key

app/models/person.rb

@@ -2,9 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require 'uri'
-require Rails.root.join('lib', 'hcard')
-
 class Person < ActiveRecord::Base
   include ROXML
   include Encryptor::Public

app/models/photo.rb

@@ -3,8 +3,6 @@
 #   the COPYRIGHT file.
 
 class Photo < ActiveRecord::Base
-  require 'carrierwave/orm/activerecord'
-
   include Diaspora::Federated::Shareable
   include Diaspora::Commentable
   include Diaspora::Shareable
@@ -129,7 +127,7 @@ class Photo < ActiveRecord::Base
   end
 
   def queue_processing_job
-    Resque.enqueue(Jobs::ProcessPhoto, self.id)
+    Workers::ProcessPhoto.perform_async(self.id)
   end
 
   def mutable?

app/models/service.rb

@@ -4,7 +4,6 @@
 
 class Service < ActiveRecord::Base
   include ActionView::Helpers::TextHelper
-  require Rails.root.join('app', 'helpers', 'markdownify_helper')
   include MarkdownifyHelper
   
   belongs_to :user
@@ -39,5 +38,5 @@ class Service < ActiveRecord::Base
   end
 
 end
-require Rails.root.join('app', 'models', 'services', 'facebook')
+require 'services/facebook'
-require Rails.root.join('app', 'models', 'services', 'twitter')
+require 'services/twitter'

app/models/services/facebook.rb

@@ -1,7 +1,5 @@
-require 'uri'
 class Services::Facebook < Service
   include Rails.application.routes.url_helpers
-  require Rails.root.join('app', 'helpers', 'markdownify_helper')
   include MarkdownifyHelper
 
   OVERRIDE_FIELDS_ON_FB_UPDATE = [:contact_id, :person_id, :request_id, :invitation_id, :photo_url, :name, :username]

app/models/services/twitter.rb

@@ -1,5 +1,3 @@
-require 'uri'
-
 class Services::Twitter < Service
   MAX_CHARACTERS = 140
   SHORTENED_URL_LENGTH = 21

app/models/status_message.rb

@@ -155,11 +155,10 @@ class StatusMessage < Post
   end
 
   def queue_gather_oembed_data
-    Resque.enqueue(Jobs::GatherOEmbedData, self.id, self.oembed_url)
+    Workers::GatherOEmbedData.perform_async(self.id, self.oembed_url)
   end
 
   def contains_oembed_url_in_text?
-    require 'uri'
     urls = URI.extract(self.raw_message, ['http', 'https'])
     self.oembed_url = urls.find{ |url| !TRUSTED_OEMBED_PROVIDERS.find(url).nil? }
   end

app/models/user.rb

@@ -2,9 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'salmon', 'salmon')
-require Rails.root.join('lib', 'postzord', 'dispatcher')
-
 class User < ActiveRecord::Base
   include Encryptor::Private
   include Connecting
@@ -167,7 +164,7 @@ class User < ActiveRecord::Base
 
   def send_reset_password_instructions
     generate_reset_password_token! if should_generate_reset_token?
-    Resque.enqueue(Jobs::ResetPassword, self.id)
+    Workers::ResetPassword.perform_async(self.id)
   end
 
   def update_user_preferences(pref_hash)
@@ -302,15 +299,15 @@ class User < ActiveRecord::Base
 
   ######### Mailer #######################
   def mail(job, *args)
-    pref = job.to_s.gsub('Jobs::Mail::', '').underscore
+    pref = job.to_s.gsub('Workers::Mail::', '').underscore
     if(self.disable_mail == false && !self.user_preferences.exists?(:email_type => pref))
-      Resque.enqueue(job, *args)
+      job.perform_async(*args)
     end
   end
 
   def mail_confirm_email
     return false if unconfirmed_email.blank?
-    Resque.enqueue(Jobs::Mail::ConfirmEmail, id)
+    Workers::Mail::ConfirmEmail.perform_async(id)
     true
   end
 

app/models/user/querying.rb

@@ -2,9 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'evil_query')
-
-
 #TODO: THIS FILE SHOULD NOT EXIST, EVIL SQL SHOULD BE ENCAPSULATED IN EvilQueries,
 #throwing all of this stuff in user violates demeter like WHOA
 module User::Querying

app/presenters/o_embed_presenter.rb

@@ -1,4 +1,3 @@
-require 'uri'
 class OEmbedPresenter
   include PostsHelper
   include ActionView::Helpers::TextHelper

app/presenters/post_presenter.rb

@@ -1,5 +1,3 @@
-require Rails.root.join('lib', 'template_picker')
-
 class PostPresenter
   attr_accessor :post, :current_user
 

app/views/admins/_admin_bar.haml

@@ -6,6 +6,5 @@
       %li= link_to t('.weekly_user_stats'), weekly_user_stats_path
       %li= link_to t('.pod_stats'), pod_stats_path
       %li= link_to t('.correlations'), correlations_path
-      - if AppConfig.admins.inline_resque_web?
+      %li= link_to t('.sidekiq_monitor'), sidekiq_path
-        %li= link_to t('.resque_overview'), resque_web_path
 	

app/workers/base.rb

@@ -0,0 +1,23 @@
+#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
+#   licensed under the Affero General Public License version 3 or later.  See
+#   the COPYRIGHT file.
+
+module Workers
+  class Base
+    include Sidekiq::Worker
+    sidekiq_options timeout: AppConfig.environment.sidekiq.timeout.to_i,
+                    backtrace: ((bt = AppConfig.environment.sidekiq.backtrace.get) && bt.to_i),
+                    retry:  AppConfig.environment.sidekiq.retry.to_i
+
+    # In the long term we need to eliminate the cause of these
+    def suppress_annoying_errors(&block)
+      yield
+    rescue Diaspora::ContactRequiredUnlessRequest,
+           Diaspora::RelayableObjectWithoutParent => e
+      Rails.logger.info("error on receive: #{e.class}")
+    rescue ActiveRecord::RecordInvalid => e
+      Rails.logger.info("failed to save received object: #{e.record.errors.full_messages}")
+      raise e unless e.message.match(/already been taken/)
+    end
+  end
+end

app/workers/deferred_dispatch.rb

@@ -2,11 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class DeferredDispatch < Base
-    @queue = :dispatch
+    sidekiq_options queue: :dispatch
 
-    def self.perform(user_id, object_class_name, object_id, opts)
+    def perform(user_id, object_class_name, object_id, opts)
       user = User.find(user_id)
       object = object_class_name.constantize.find(object_id)
       opts = HashWithIndifferentAccess.new(opts)

app/workers/delete_account.rb

@@ -3,10 +3,11 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   class DeleteAccount < Base
-    @queue = :delete_account
+    sidekiq_options queue: :delete_account
-    def self.perform(account_deletion_id)
+    
+    def perform(account_deletion_id)
       account_deletion = AccountDeletion.find(account_deletion_id)
       account_deletion.perform!
     end

app/workers/delete_post_from_service.rb

@@ -2,11 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 #
-module Jobs
+module Workers
   class DeletePostFromService < Base
-    @queue = :http_service
+    sidekiq_options queue: :http_service
 
-    def self.perform(service_id, service_post_id)
+    def perform(service_id, service_post_id)
       service = Service.find_by_id(service_id)
       service.delete_post(service_post_id)
     end

app/workers/fetch_profile_photo.rb

@@ -3,10 +3,11 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   class FetchProfilePhoto < Base
-    @queue = :photos
+    sidekiq_options queue: :photos
-    def self.perform(user_id, service_id, fallback_image_url = nil)
+
+    def perform(user_id, service_id, fallback_image_url = nil)
       service = Service.find(service_id)
 
       image_url = service.profile_photo_url

app/workers/fetch_public_posts.rb

@@ -2,14 +2,12 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class FetchPublicPosts < Base
-    @queue = :http_service
+    sidekiq_options queue: :http_service
 
-    def self.perform(diaspora_id)
+    def perform(diaspora_id)
-      require Rails.root.join('lib','diaspora','fetcher','public')
+      Diaspora::Fetcher::Public.new.fetch!(diaspora_id)
-
-      PublicFetcher.new.fetch!(diaspora_id)
     end
   end
 end

app/workers/fetch_webfinger.rb

@@ -2,15 +2,15 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class FetchWebfinger < Base
-    @queue = :socket_webfinger
+    sidekiq_options queue: :socket_webfinger
 
-    def self.perform(account)
+    def perform(account)
       person = Webfinger.new(account).fetch
 
       # also, schedule to fetch a few public posts from that person
-      Resque.enqueue(Jobs::FetchPublicPosts, person.diaspora_handle) unless person.nil?
+      Workers::FetchPublicPosts.perform_async(person.diaspora_handle) unless person.nil?
     end
   end
 end

app/workers/gather_o_embed_data.rb

@@ -0,0 +1,22 @@
+#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
+#   licensed under the Affero General Public License version 3 or later.  See
+#   the COPYRIGHT file.
+#
+
+module Workers
+  class GatherOEmbedData < Base
+    sidekiq_options queue: :http_service
+
+    def perform(post_id, url, retry_count=1)
+      post = Post.find(post_id)
+      post.o_embed_cache = OEmbedCache.find_or_create_by_url(url)
+      post.save
+    rescue ActiveRecord::RecordNotFound
+      # User created a post and deleted it right afterwards before we
+      # we had a chance to run the job.
+      # On the other hand sometimes the job runs before the Post is
+      # fully persisted. So we just reduce the amount of retries.
+      GatherOEmbedData.perform_in(1.minute, post_id, url, retry_count+1) unless retry_count > 3
+    end
+  end
+end

app/workers/http_multi.rb

@@ -2,17 +2,13 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require 'uri'
+module Workers
-require Rails.root.join('lib', 'hydra_wrapper')
-
-module Jobs
   class HttpMulti < Base
-
+    sidekiq_options queue: :http
-    @queue = :http
 
     MAX_RETRIES = 3
 
-    def self.perform(user_id, encoded_object_xml, person_ids, dispatcher_class_as_string, retry_count=0)
+    def perform(user_id, encoded_object_xml, person_ids, dispatcher_class_as_string, retry_count=0)
       user = User.find(user_id)
       people = Person.where(:id => person_ids)
 
@@ -24,7 +20,7 @@ module Jobs
 
       unless hydra.failed_people.empty?
         if retry_count < MAX_RETRIES
-          Resque.enqueue(Jobs::HttpMulti, user_id, encoded_object_xml, hydra.failed_people, dispatcher_class_as_string, retry_count + 1 )
+          Workers::HttpMulti.perform_in(1.hour, user_id, encoded_object_xml, hydra.failed_people, dispatcher_class_as_string, retry_count + 1)
         else
           Rails.logger.info("event=http_multi_abandon sender_id=#{user_id} failed_recipient_ids='[#{person_ids.join(', ')}] '")
         end

app/workers/mail/also_commented.rb

@@ -1,8 +1,9 @@
-module Jobs
+module Workers
   module Mail
     class AlsoCommented < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, sender_id, comment_id)
+
+      def perform(recipient_id, sender_id, comment_id)
         if email = Notifier.also_commented(recipient_id, sender_id, comment_id)
           email.deliver
         end

app/workers/mail/comment_on_post.rb

@@ -1,8 +1,9 @@
-module Jobs
+module Workers
   module Mail
     class CommentOnPost < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, sender_id, comment_id)
+
+      def perform(recipient_id, sender_id, comment_id)
         Notifier.comment_on_post(recipient_id, sender_id, comment_id).deliver
       end
     end

app/workers/mail/confirm_email.rb

@@ -1,8 +1,9 @@
-module Jobs
+module Workers
   module Mail
     class ConfirmEmail < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(user_id)
+      
+      def perform(user_id)
         Notifier.confirm_email(user_id).deliver
       end
     end

app/workers/mail/invite_user_by_email.rb

@@ -2,12 +2,12 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-
+module Workers
-module Jobs
   module Mail
     class InviteUserByEmail < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(invite_id)
+      
+      def perform(invite_id)
         invite = Invitation.find(invite_id)
         I18n.with_locale(invite.language) do
           invite.send!

app/workers/mail/liked.rb

@@ -1,8 +1,9 @@
-module Jobs
+module Workers
   module Mail
     class Liked < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, sender_id, like_id)
+      
+      def perform(recipient_id, sender_id, like_id)
         Notifier.liked(recipient_id, sender_id, like_id).deliver
       end
     end

app/workers/mail/mentioned.rb

@@ -3,11 +3,12 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   module Mail
     class Mentioned < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, actor_id, target_id)
+      
+      def perform(recipient_id, actor_id, target_id)
         Notifier.mentioned( recipient_id, actor_id, target_id).deliver
       end
     end

app/workers/mail/private_message.rb

@@ -3,11 +3,12 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   module Mail
     class PrivateMessage < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, actor_id, target_id)
+      
+      def perform(recipient_id, actor_id, target_id)
         Notifier.private_message( recipient_id, actor_id, target_id).deliver
       end
     end

app/workers/mail/reshared.rb

@@ -1,8 +1,9 @@
-module Jobs
+module Workers
   module Mail
     class Reshared < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, sender_id, reshare_id)
+      
+      def perform(recipient_id, sender_id, reshare_id)
         Notifier.reshared(recipient_id, sender_id, reshare_id).deliver
       end
     end

app/workers/mail/started_sharing.rb

@@ -3,11 +3,12 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   module Mail
     class StartedSharing < Base
-      @queue = :mail
+      sidekiq_options queue: :mail
-      def self.perform(recipient_id, sender_id, target_id)
+      
+      def perform(recipient_id, sender_id, target_id)
         Notifier.started_sharing(recipient_id, sender_id).deliver
       end
     end

app/workers/notify_local_users.rb

@@ -2,13 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class NotifyLocalUsers < Base
-    @queue = :receive_local
+    sidekiq_options queue: :receive_local
 
-    require Rails.root.join('app', 'models', 'notification')
+    def perform(user_ids, object_klass, object_id, person_id)
-
-    def self.perform(user_ids, object_klass, object_id, person_id)
 
       object = object_klass.constantize.find_by_id(object_id)
 

app/workers/post_to_service.rb

@@ -2,11 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 #
-module Jobs
+module Workers
   class PostToService < Base
-    @queue = :http_service
+    sidekiq_options queue: :http_service
 
-    def self.perform(service_id, post_id, url)
+    def perform(service_id, post_id, url)
       service = Service.find_by_id(service_id)
       post = Post.find_by_id(post_id)
       service.post(post, url)

app/workers/process_photo.rb

@@ -3,10 +3,11 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   class ProcessPhoto < Base
-    @queue = :photos
+    sidekiq_options queue: :photos
-    def self.perform(id)
+
+    def perform(id)
       photo = Photo.find(id)
       unprocessed_image = photo.unprocessed_image
 

app/workers/publish_to_hub.rb

@@ -2,12 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class PublishToHub < Base
-    @queue = :http_service
+    sidekiq_options queue: :http_service
 
-    def self.perform(sender_public_url)
+    def perform(sender_public_url)
-      require Rails.root.join('lib', 'pubsubhubbub')
       atom_url = sender_public_url + '.atom'
       Pubsubhubbub.new(AppConfig.environment.pubsub_server.get).publish(atom_url)
     end

app/workers/receive.rb

@@ -0,0 +1,19 @@
+#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
+#   licensed under the Affero General Public License version 3 or later.  See
+#   the COPYRIGHT file.
+
+
+module Workers
+  class Receive < Base
+    sidekiq_options queue: :receive
+
+    def perform(user_id, xml, salmon_author_id)
+      suppress_annoying_errors do
+        user = User.find(user_id)
+        salmon_author = Person.find(salmon_author_id)
+        zord = Postzord::Receiver::Private.new(user, :person => salmon_author)
+        zord.parse_and_receive(xml)
+      end
+    end
+  end
+end

app/workers/receive_encrypted_salmon.rb

@@ -3,12 +3,11 @@
 #   the COPYRIGHT file.
 
 
-require Rails.root.join('lib', 'postzord', 'receiver', 'private')
+module Workers
-module Jobs
   class ReceiveEncryptedSalmon < Base
-    @queue = :receive_salmon
+    sidekiq_options queue: :receive_salmon
 
-    def self.perform(user_id, xml)
+    def perform(user_id, xml)
       suppress_annoying_errors do
         user = User.find(user_id)
         zord = Postzord::Receiver::Private.new(user, :salmon_xml => xml)

app/workers/receive_local_batch.rb

@@ -2,15 +2,11 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join('lib', 'postzord', 'receiver', 'private')
+module Workers
-require Rails.root.join('lib', 'postzord', 'receiver', 'local_batch')
-
-module Jobs
   class ReceiveLocalBatch < Base
+    sidekiq_options queue: :receive
 
-    @queue = :receive
+    def perform(object_class_string, object_id, recipient_user_ids)
-
-    def self.perform(object_class_string, object_id, recipient_user_ids)
       object = object_class_string.constantize.find(object_id)
       receiver = Postzord::Receiver::LocalBatch.new(object, recipient_user_ids)
       receiver.perform!

app/workers/receive_unencrypted_salmon.rb

@@ -0,0 +1,21 @@
+#   Copyright (c) 2010-2011, Diaspora Inc.  This file is
+#   licensed under the Affero General Public License version 3 or later.  See
+#   the COPYRIGHT file.
+
+module Workers
+  class ReceiveUnencryptedSalmon < Base
+    sidekiq_options queue: :receive
+
+    def perform(xml)
+      suppress_annoying_errors do
+        begin
+          receiver = Postzord::Receiver::Public.new(xml)
+          receiver.perform!
+        rescue => e
+          FEDERATION_LOGGER.info(e.message)
+          raise e
+        end
+      end
+    end
+  end
+end

app/workers/resend_invitation.rb

@@ -3,10 +3,11 @@
 #   the COPYRIGHT file.
 
 
-module Jobs
+module Workers
   class ResendInvitation < Base
-    @queue = :mail
+    sidekiq_options queue: :mail
-    def self.perform(invitation_id)
+    
+    def perform(invitation_id)
       inv = Invitation.find(invitation_id)
       inv.resend
     end

app/workers/reset_password.rb

@@ -1,8 +1,8 @@
-module Jobs
+module Workers
   class ResetPassword < Base
-    @queue = :mail
+    sidekiq_options queue: :mail
 
-    def self.perform(user_id)
+    def perform(user_id)
       user = User.find(user_id)
       ::Devise.mailer.reset_password_instructions(user).deliver
     end

config/application.rb

@@ -32,8 +32,8 @@ module Diaspora
     # -- all .rb files in that directory are automatically loaded.
 
     # Add additional load paths for your own custom dirs
-     config.autoload_paths += %W(#{config.root}/app/presenters)
+     config.autoload_paths      += %W{#{config.root}/app/presenters}
-     config.autoload_paths += %W(#{config.root}/lib)
+     config.autoload_once_paths += %W{#{config.root}/lib}
 
     # Only load the plugins named here, in the order given (default is alphabetical).
     # :all can be used as a placeholder for all plugins not explicitly named

config/database.yml.example

@@ -18,11 +18,14 @@ postgres: &postgres
 
 # Comment the the mysql line and uncomment the postgres line
 # if you want to use postgres
-choose: &choose
+common: &common
   # Choose one of the following
   <<: *mysql
   #<<: *postgres
 
+  # Should match environment.sidekiq.concurrency
+  #pool: 25
+  
 ##################################################
 #### CONFIGURE ABOVE #############################
 ##################################################
@@ -32,20 +35,20 @@ choose: &choose
 postgres_travis: &postgres_travis
   adapter: postgresql
   username: postgres
-common: &common
+combined: &combined
-  <<: *choose
-development:
   <<: *common
+development:
+  <<: *combined
   database: diaspora_development
 production:
-  <<: *common
+  <<: *combined
   database: diaspora_production
 test:
-  <<: *common
+  <<: *combined
   database: "diaspora_test"
 integration1:
-  <<: *common
+  <<: *combined
   database: diaspora_integration1
 integration2:
-  <<: *common
+  <<: *combined
   database: diaspora_integration2

config/defaults.yml

@@ -12,6 +12,13 @@ defaults:
     redis:
     require_ssl: true
     single_process_mode: false
+    sidekiq:
+      namespace: "diaspora"
+      concurrency: 5
+      retry: 10
+      timeout: 900
+      backtrace: 15
+      log: "log/sidekiq.log"
     s3:
       enable: false
       key:
@@ -32,8 +39,7 @@ defaults:
     stdout_log:
     database: 'mysql'
     unicorn_worker: 2
-    embed_resque_worker: false
+    embed_sidekiq_worker: false
-    resque_workers: 1
   privacy:
     jquery_cdn: true
     google_analytics_key:
@@ -88,7 +94,6 @@ defaults:
   admins:
     account:
     podmin_email:
-    inline_resque_web: true
 
 development:
   environment:
@@ -123,12 +128,14 @@ test:
 integration1:
   environment:
     url: "http://localhost:45789/"
+    single_process_mode: true
     assets:
       serve: true
     require_ssl: false
 integration2:
   environment:
     url: "http://localhost:34658/"
+    single_process_mode: true
     assets:
       serve: true
     require_ssl: false

config/diaspora.yml.example

@@ -67,6 +67,38 @@ configuration: ## Section
     ## So this is higly unrecommended for production setups.
     #single_process_mode: true
 
+    ## Sidekiq - background processing
+    sidekiq: ## Section
+      ## Number of parallel threads Sidekiq uses
+      ## If you touch this please set the pool setting
+      ## in your database.yml to a value that's at minimum
+      ## close to this! The default value is 5 but you can safely
+      ## increase it to 25 and more on a medium sized pod.
+      ## This applies per started Sidekiq worker, so if you set it to
+      ## 25 and start two workers you'll process up to 50 jobs in parallel.
+      #concurrency: 25
+
+      ## Number of times a job is retried
+      ## There's an exponential backoff, if you set this too
+      ## high you might get too many jobs in the queue
+      ## Set this to false to disable it completely 
+      #retry: 10
+
+      ## Time in seconds before a job is killed
+      #timeout: 900
+
+      ## Namespace to use in Redis, useful if you need to run
+      ## multiple instances of Diaspora using the same Redis instance
+      #namespace: "diaspora"
+
+      ## Lines of backtrace that is stored on failure
+      ## Set this to false if you're not interested in this data to
+      ## reduce memory usage (and log size)
+      #backtrace: 15
+
+      ## Log file for Sidekiq
+      #log: "log/sidekiq.log"
+    
     ## Use Amazon S3 instead of your local filesystem
     ## to handle uploaded pictures.
     s3: ## Section
@@ -124,12 +156,9 @@ configuration: ## Section
     ## you have many users
     #unicorn_worker: 2
     
-    ## Embed a resque worker inside the unicorn process, useful for
+    ## Embed a Sidekiq worker inside the unicorn process, useful for
     ## minimal Heroku setups
-    #embed_resque_worker: true
+    #embed_sidekiq_worker: true
-    
-    ## Number of resque workers to start
-    #resque_workers: 1
   
   ## Settings probably affecting the privacy of your users 
   privacy: ## Section
@@ -294,11 +323,6 @@ configuration: ## Section
     ## E-Mail address users can contact the administrator
     #podmin_email: 'podmin@example.org'
   
-    ## Resque is the background processing system used by Diaspora
-    ## Resque web is an admin tool for it. This settings decides whether
-    ## or not to inline it into Diaspora.
-    #inline_resque_web: true
-  
 ## Here you can make overides to settings defined above if you need
 ## to have them different in different environments.
 production: ## Section

config/environment.rb

@@ -13,7 +13,6 @@ end
 
 # Load the rails application
 require Pathname.new(__FILE__).dirname.expand_path.join('application')
-require Rails.root.join("lib", "exceptions")
 
 # Load configuration system early 
 require Rails.root.join('config', 'load_config')
@@ -27,7 +26,6 @@ USERNAME_BLACKLIST = ['admin', 'administrator', 'hostmaster', 'info', 'postmaste
 
 # Initialize the rails application
 Diaspora::Application.initialize!
-require Rails.root.join('lib', 'federation_logger')
 
 # allow token auth only for posting activitystream photos
 module Devise

config/initializers/acts_as_taggable_on.rb

@@ -1,3 +1,2 @@
-
 ActsAsTaggableOn.force_lowercase = true
-
+require Rails.root.join("app", "models", "acts_as_taggable_on", "tag")

config/initializers/load_libraries.rb

@@ -0,0 +1,36 @@
+# Stdlib
+require 'cgi'
+require 'uri'
+
+# Not auto required gems
+require 'active_support/base64'
+require 'builder/xchar'
+require 'carrierwave/orm/activerecord'
+require 'erb'
+require 'redcarpet/render_strip'
+require 'typhoeus'
+
+# Presenters
+require 'post_presenter'
+
+# Helpers
+require 'markdownify_helper'
+
+# Our libs
+require 'collect_user_photos'
+require 'diaspora'
+require 'email_inviter'
+require 'evil_query'
+require 'federation_logger'
+require 'h_card'
+require 'hydra_wrapper'
+require 'postzord'
+require 'publisher'
+require 'pubsubhubbub'
+require 'salmon'
+require 'statistics'
+require 'stream'
+require 'template_picker'
+require 'webfinger'
+require 'webfinger_profile'
+

config/initializers/load_mail_jobs.rb

@@ -1 +0,0 @@
-Dir[Rails.root.join('app', 'models', 'jobs', 'mail', '*.rb')].each { |file| require file }

config/initializers/resque.rb

@@ -1,36 +0,0 @@
-require 'resque'
-
-Resque::Plugins::Timeout.timeout = 300
-
-if !AppConfig.environment.single_process_mode?
-  Resque.redis = AppConfig.get_redis_instance
-end
-
-# Single process-mode hooks using Resque.inline
-if AppConfig.environment.single_process_mode?
-  if Rails.env == 'production'
-    puts "WARNING: You are running Diaspora in production without Resque"
-    puts "  workers turned on.  Please set single_process_mode to false in"
-    puts "  config/diaspora.yml."
-  end
-  Resque.inline = true
-end
-
-if AppConfig.admins.monitoring.airbrake_api_key.present?
-  require 'resque/failure/multiple'
-  require 'resque/failure/airbrake'
-  require 'resque/failure/redis'
-  Resque::Failure::Airbrake.configure do |config|
-    config.api_key = AppConfig.admins.monitoring.airbrake_api_key
-    config.secure = true
-  end
-  Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
-  Resque::Failure.backend = Resque::Failure::Multiple
-end
-
-
-if AppConfig.admins.inline_resque_web?
-  require 'resque/server'
-  require Rails.root.join('lib', 'admin_rack')
-  Resque::Server.use AdminRack
-end

config/initializers/sidekiq.rb

@@ -0,0 +1,50 @@
+require 'sidekiq_middlewares'
+
+# Single process-mode
+if AppConfig.environment.single_process_mode? && Rails.env != "test"
+  if Rails.env == 'production'
+    puts "WARNING: You are running Diaspora in production without Sidekiq"
+    puts "         workers turned on.  Please set single_process_mode to false in"
+    puts "         config/diaspora.yml."
+  end
+  require 'sidekiq/testing/inline'
+end
+
+
+Sidekiq.configure_server do |config|
+  config.redis = AppConfig.get_redis_options
+
+  config.options = config.options.merge({
+    concurrency: AppConfig.environment.sidekiq.concurrency.to_i,
+    queues: %w{
+      socket_webfinger
+      photos
+      http_service
+      dispatch
+      mail
+      delete_account
+      receive_local
+      receive
+      receive_salmon
+      http
+      default
+    }
+  })
+
+  config.server_middleware do |chain|
+    chain.add SidekiqMiddlewares::CleanAndShortBacktraces
+  end
+
+  Sidekiq::Logging.initialize_logger AppConfig.sidekiq_log unless AppConfig.heroku?
+
+  # Set connection pool on Heroku
+  database_url = ENV['DATABASE_URL']
+  if(database_url)
+    ENV['DATABASE_URL'] = "#{database_url}?pool=#{AppConfig.environment.sidekiq.concurrency.get}"
+    ActiveRecord::Base.establish_connection
+  end
+end
+
+Sidekiq.configure_client do |config|
+  config.redis = AppConfig.get_redis_options
+end

config/locales/diaspora/en.yml

@@ -89,7 +89,7 @@ en:
       weekly_user_stats: "Weekly User Stats"
       pod_stats: "Pod Stats"
       correlations: "Correlations"
-      resque_overview: "Resque Overview"
+      sidekiq_monitor: "Sidekiq monitor"
     correlations:
       correlations_count: "Correlations with Sign In Count:"
     user_search:

config/routes.rb

@@ -2,11 +2,17 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
+require 'sidekiq/web'
+
 Diaspora::Application.routes.draw do
   if Rails.env.production?
     mount RailsAdmin::Engine => '/admin_panel', :as => 'rails_admin'
   end
 
+  constraints ->(req) { req.env["warden"].authenticate?(scope: :user) &&
+                        req.env['warden'].user.admin? } do
+    mount Sidekiq::Web => '/sidekiq', :as => 'sidekiq'
+  end
 
   get "/atom.xml" => redirect('http://blog.diasporafoundation.org/feed/atom') #too many stupid redirects :()
 
@@ -206,11 +212,6 @@ Diaspora::Application.routes.draw do
   #Protocol Url
   get 'protocol' => redirect("https://github.com/diaspora/diaspora/wiki/Diaspora%27s-federation-protocol")
 
-  # Resque web
-  if AppConfig.admins.inline_resque_web?
-    mount Resque::Server.new, :at => '/resque-jobs', :as => "resque_web"
-  end
-
   # Startpage
   root :to => 'home#show'
 end

config/unicorn.rb

@@ -15,7 +15,7 @@ preload_app true
 # How long to wait before killing an unresponsive worker
 timeout 30
 
-@resque_pid = nil
+@sidekiq_pid = nil
 
 #pid '/var/run/diaspora/diaspora.pid'
 #listen '/var/run/diaspora/diaspora.sock', :backlog => 2048
@@ -33,14 +33,12 @@ before_fork do |server, worker|
   ActiveRecord::Base.connection.disconnect!
 
   # disconnect redis if in use
-  if !AppConfig.single_process_mode?
+  unless AppConfig.single_process_mode?
-    Resque.redis.client.disconnect
+    Sidekiq.redis {|redis| redis.client.disconnect }
   end
   
-  if AppConfig.server.embed_resque_worker?
+  if AppConfig.server.embed_sidekiq_worker?
-    # Clean up Resque workers killed by previous deploys/restarts
+    @sidekiq_pid ||= spawn('bundle exec sidekiq')
-    Resque.workers.each { |w| w.unregister_worker }
-    @resque_pid ||= spawn('bundle exec rake resque:work QUEUES=*')
   end
 
   old_pid = '/var/run/diaspora/diaspora.pid.oldbin'
@@ -58,9 +56,8 @@ after_fork do |server, worker|
   # If using preload_app, enable this line
   ActiveRecord::Base.establish_connection
 
-  # copy pasta from resque.rb because i'm a bad person
+  unless AppConfig.environment.single_process_mode?
-  if !AppConfig.environment.single_process_mode?
+    Sidekiq.redis = AppConfig.get_redis_options
-    Resque.redis = AppConfig.get_redis_instance
   end
 
   # Enable this line to have the workers run as different user/group

db/seeds.rb

@@ -46,7 +46,7 @@ Role.add_admin(bob.person)
 puts "done!"
 
 
-require Rails.root.join('spec', 'support', 'fake_resque')
+require 'sidekiq/testing/inline'
 require Rails.root.join('spec', 'support', 'user_methods')
 
 print "Seeding post data..."

features/step_definitions/debug_steps.rb

@@ -1,6 +1,6 @@
 module DebuggingCukeHelpers
   def start_debugging
-    require 'ruby-debug'
+    require 'debug'
     debugger
     true
   end

features/step_definitions/web_steps.rb

@@ -4,8 +4,6 @@
 # instead of editing this one. Cucumber will automatically load all features/**/*.rb
 # files.
 
-require 'uri'
-require 'cgi'
 require File.expand_path(File.join(File.dirname(__FILE__), "..", "support", "paths"))
 
 module WithinHelpers

features/support/env.rb

@@ -46,16 +46,16 @@ prefork = proc do
   require File.join(File.dirname(__FILE__), "integration_sessions_controller")
   require File.join(File.dirname(__FILE__), "poor_mans_webmock")
 
+  require 'sidekiq/testing/inline'
+
   require Rails.root.join('spec', 'helper_methods')
+  require Rails.root.join('spec', 'support', 'inlined_jobs')
   require Rails.root.join('spec', 'support', 'user_methods')
   include HelperMethods
 
   # require 'webmock/cucumber'
   # WebMock.disable_net_connect!(:allow_localhost => true)
 
-  require Rails.root.join('spec', 'support', 'fake_resque')
-
-  require File.join(File.dirname(__FILE__), 'run_resque_in_process')
 
   #hax to get rubymine to run spork, set RUBYMINE_HOME in your .bash_profile
   if ENV["RUBYMINE_HOME"]

features/support/poor_mans_webmock.rb

@@ -2,31 +2,27 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-module Jobs
+module Workers
   class PublishToHub < Base
-    @queue = :http_service
+    def perform(sender_public_url)
-    def self.perform(sender_public_url)
       # don't publish to pubsubhubbub in cucumber
     end
   end
 
   class HttpMulti < Base
-    @queue = :http
+    def perform(user_id, enc_object_xml, person_ids, retry_count=0)
-    def self.perform(user_id, enc_object_xml, person_ids, retry_count=0)
       # don't federate in cucumber
     end
   end
 
   class HttpPost < Base
-    @queue = :http
+    def perform(url, body, tries_remaining = NUM_TRIES)
-    def self.perform(url, body, tries_remaining = NUM_TRIES)
       # don't post to outside services in cucumber
     end
   end
 
   class PostToService < Base
-    @queue = :http_service
+    def perform(service_id, post_id, url)
-    def self.perform(service_id, post_id, url)
       # don't post to services in cucumber
     end
   end

features/support/run_resque_in_process.rb

@@ -1,5 +0,0 @@
-module Resque
-  def enqueue(klass, *args)
-    klass.send(:perform, *args)
-  end
-end

lib/admin_rack.rb

@@ -1,19 +0,0 @@
-class AdminRack
-  def initialize(app)
-    @app = app
-  end
-
-  def call(env)
-    user = env['warden'].authenticate(:scope => :user)
-    if user && user.admin?
-      @app.call(env)
-    else
-      [307, {"Location" => '/'}, self]
-    end
-  end
-
-  def each(&block)
-  end
-end
-
-

lib/configuration_methods.rb

@@ -74,7 +74,7 @@ module Configuration
       (git_revision || version)[0..8]
     end
     
-    def get_redis_instance
+    def get_redis_options
       if redistogo_url.present?
         $stderr.puts "WARNING: using the REDISTOGO_URL environment variable is deprecated, please use REDIS_URL now."
         ENV['REDIS_URL'] = redistogo_url
@@ -85,15 +85,23 @@ module Configuration
       redis_url = ENV['REDIS_URL'] || environment.redis.get
       
       if ENV['RAILS_ENV']== 'integration2'
-        redis_options = { :host => 'localhost', :port => 6380 }
+        redis_options[:url] = "redis://localhost:6380"
       elsif redis_url.present?
         unless redis_url.start_with?("redis://") || redis_url.start_with?("unix:///")
           $stderr.puts "WARNING: Your redis url (#{redis_url}) doesn't start with redis:// or unix:///"
         end
-        redis_options = { :url => redis_url }
+        redis_options[:url] = redis_url
       end
       
-      Redis.new(redis_options.merge(:thread_safe => true))
+      redis_options[:namespace] = AppConfig.environment.sidekiq_namespace.get
+      
+      redis_options
+    end
+    
+    def sidekiq_log
+      path = Pathname.new environment.sidekiq.log.get
+      path = Rails.root.join(path) unless pathname.absolute?
+      path.to_s
     end
 
     private

lib/diaspora.rb

@@ -3,5 +3,9 @@
 #   the COPYRIGHT file.
 
 module Diaspora
-  autoload :Parser
+  require 'diaspora/exceptions'
+  require 'diaspora/parser'
+  require 'diaspora/fetcher'
+  require 'diaspora/markdownify'
+  require 'diaspora/exporter'
 end

lib/diaspora/exceptions.rb

@@ -17,4 +17,17 @@ module Diaspora
   class NotMine < StandardError
   end
   
+
+  # Received a message without having a contact
+  class ContactRequiredUnlessRequest < StandardError
+  end
+
+  # Got a relayable (comment, like etc.) without having the parent
+  class RelayableObjectWithoutParent < StandardError
+  end
+
+  # After building an object the author doesn't match the one in the
+  # original XML message
+  class AuthorXMLAuthorMismatch < StandardError
+  end
 end

lib/diaspora/federated/base.rb

@@ -9,8 +9,6 @@
 
 #including this module lets you federate an object at the most basic of level
 
-require 'builder/xchar'
-
 module Diaspora
   module Federated
     module Base 

lib/diaspora/fetcher.rb

@@ -0,0 +1,5 @@
+module Diaspora
+  module Fetcher
+    require 'diaspora/fetcher/public'
+  end
+end

lib/diaspora/fetcher/public.rb

@@ -1,8 +1,7 @@
 #   Copyright (c) 2010-2012, Diaspora Inc.  This file is
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
-
+module Diaspora; module Fetcher; class Public
-class PublicFetcher
 
   # various states that can be assigned to a person to describe where
   # in the process of fetching their public posts we're currently at
@@ -23,11 +22,11 @@ class PublicFetcher
     begin
       retrieve_and_process_posts
     rescue => e
-      set_fetch_status PublicFetcher::Status_Failed
+      set_fetch_status Public::Status_Failed
       raise e
     end
 
-    set_fetch_status PublicFetcher::Status_Done
+    set_fetch_status Public::Status_Done
   end
 
   private
@@ -35,20 +34,20 @@ class PublicFetcher
     # if it is reasonable to do so, and that they have not been fetched already
     def qualifies_for_fetching?
       raise ActiveRecord::RecordNotFound unless @person.present?
-      return false if @person.fetch_status == PublicFetcher::Status_Unfetchable
+      return false if @person.fetch_status == Public::Status_Unfetchable
 
       # local users don't need to be fetched
       if @person.local?
-        set_fetch_status PublicFetcher::Status_Unfetchable
+        set_fetch_status Public::Status_Unfetchable
         return false
       end
 
       # this record is already being worked on
-      return false if @person.fetch_status > PublicFetcher::Status_Initial
+      return false if @person.fetch_status > Public::Status_Initial
 
       # ok, let's go
       @person.remote? &&
-      @person.fetch_status == PublicFetcher::Status_Initial
+      @person.fetch_status == Public::Status_Initial
     end
 
     # call the methods to fetch and process the public posts for the person
@@ -72,24 +71,19 @@ class PublicFetcher
     # fetch the public posts of the person from their server and save the
     # JSON response to `@data`
     def retrieve_posts
-      set_fetch_status PublicFetcher::Status_Running
+      set_fetch_status Public::Status_Running
 
       FEDERATION_LOGGER.info "fetching public posts for #{@person.diaspora_handle}"
 
-      conn = Faraday.new(:url => @person.url) do |c|
+      resp = Faraday.get("#{@person.url}people/#{@person.guid}") do |req|
-        c.request :json
+        req.headers[:accept] = 'application/json'
-        c.response :json
+        req.headers[:user_agent] = 'diaspora-fetcher'
-        c.adapter :net_http
       end
-      conn.headers[:user_agent] = 'diaspora-fetcher'
-      conn.headers[:accept] = 'application/json'
-
-      resp = conn.get "/people/#{@person.guid}"
 
       FEDERATION_LOGGER.debug resp.body.to_s[0..250]
 
-      @data = resp.body
+      @data = JSON.parse resp.body
-      set_fetch_status PublicFetcher::Status_Fetched
+      set_fetch_status Public::Status_Fetched
     end
 
     # process the public posts that were previously fetched with `retrieve_posts`
@@ -124,7 +118,7 @@ class PublicFetcher
         StatusMessage.set_callback :create, :set_guid
 
       end
-      set_fetch_status PublicFetcher::Status_Processed
+      set_fetch_status Public::Status_Processed
     end
 
     # set and save the fetch status for the current person
@@ -182,4 +176,4 @@ class PublicFetcher
 
       type_ok
     end
-end
+end; end; end

lib/diaspora/markdownify.rb

@@ -1,5 +1,3 @@
-require 'erb'
-
 module Diaspora
   module Markdownify
     class HTML < Redcarpet::Render::HTML

lib/diaspora/markdownify_email.rb

@@ -1,5 +1,3 @@
-require Rails.root.join("app", "models", "acts_as_taggable_on", "tag")
-
 module Diaspora
   module Markdownify
     class Email < Redcarpet::Render::HTML

lib/diaspora/parser.rb

@@ -13,7 +13,7 @@ module Diaspora
       rescue NameError => e
         # A pods is trying to federate an object we don't recognize.
         # i.e. their codebase is different from ours.  Quietly discard
-        # so that no Resque job failure is created
+        # so that no job failure is created
         nil
       end
     end

lib/diaspora/taggable.rb

@@ -2,8 +2,6 @@
 #   licensed under the Affero General Public License version 3 or later.  See
 #   the COPYRIGHT file.
 
-require Rails.root.join("app", "models", "acts_as_taggable_on", "tag")
-
 module Diaspora
   module Taggable
     def self.included(model)