nocebo/diaspora
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Bye Resque. Ohai Sidekiq.

Browse source 
* Dropped all references to Resque
* Moved all jobs under app/workers since that's the Sidekiq convention
* Renamed Jobs module to Worker to match new location
* Adapted all jobs to Sidekiq
* Replaced all enqueue calls with perform_async
* Dropped Resque hacks from specs and features, replaced with
  sidekig/testing in RSpec and sidekig/testing/inline in Cucumber
* Updated scripts to start a Sidekiq server
* Inline Sidekiq sinatra app
* Let Sidekiq create the actual Redis instance
* Workaround already initialized constant warnings in service models
* Resolved ToDo in one job definition by creating proper exception clases
  for some errors in receiving posts
* Added sidekiq section to configuration to make it completly
  configurable to the user
* Add Sidekiq middleware for clean backtraces
* Delay HttpMulti retry to give offline pods a chance to come back up
* Do not retry on GUID already taken and alike errors
* Be graceful about deleted posts in GatherOEmbedData
This commit is contained in:
Jonne Haß 2013-02-19 09:34:41 +01:00
parent 3fc3b249e7
commit 79a79d65d6
117 changed files with 688 additions and 556 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.foreman
View file

@ -1,2 +1,2 @@
port: 3000
formation: web=1,worker=0
formation: web=1,sidekiq=0

106
Changelog.md
View file

@ -1,5 +1,102 @@
# Head
## Refactor
### Replaced Resque with Sidekiq - Migration guide - [#3993](https://github.com/diaspora/diaspora/pull/3993)
We replaced our queue system with Sidekiq. You might know that Resque needs Redis.
Sidekiq does too, so don't remove it, it's still required. Sidekiq uses a threaded
model so you'll need far less processes than with Resque to do the same amount
of work.
To update do the following:
1. Before updating (even before the `git pull`!) stop your application
server (Unicorn by default, started through Foreman).
2. In case you did already run `git pull` checkout v0.0.3.2:
```
git fetch origin
git checkout v0.0.3.2
bundle
```
3. Start Resque web (you'll need temporary access to port 5678, check
your Firewall if needed!):
```
bundle exec resque-web
```
In case you need it you can adjust the port with the `-p` flag.
4. One last time, start a Resque worker:
```
RAILS_ENV=production QUEUE=* bundle exec rake resque:work
```
Visit Resque web via http://your_host:5678, wait until all queues but the
failed one are empty (show 0 jobs).
5. Kill the Resque worker by hitting Ctrl+C. Kill Resque web with:
```
bundle exec resque-web -k
```
Don't forget to close the port on the Firewall again, if you had to open it.
6. In case you needed to do step 2., run:
```
git checkout master
bundle
```
7. Proceed with the update as normal (migrate database, precompile assets).
8. Before starting Diaspora again ensure that you reviewed the new
`environment.sidekiq` section in `config/diaspora.yml.example` and,
if wanted, transfered it to your `config/diaspora.yml` and made any
needed changes. In particular increase the `environment.sidekiq.concurrency`
setting on any medium sized pod. If you do change that value, edit
your `config/database.yml` and add a matching `pool: n` to your database
configuration. n should be equal or higher than the amount of
threads per Sidekiq worker. This sets how many concurrent
connections to the database ActiveRecord allows.
If you aren't using `script/server` but for example passenger, you no
longer need to start a Resque worker, but a Sidekiq worker now. The
command for that is:
```
bundle exec sidekiq
```
#### Heroku
The only gotcha for Heroku single gear setups is that the setting name
to spawn a background worker from the unicorn process changed. Run
```
heroku config:remove SERVER_EMBED_RESQUE_WORKER
heroku config:set SERVER_EMBED_SIDEKIQ_WORKER=true
```
We're automatically adjusting the ActiveRecord connection pool size for you.
Larger Heroku setups should have enough expertise to figure out what to do
by them self.
### Other
* Cleaned up requires of our own libraries [#3993](https://github.com/diaspora/diaspora/pull/3993)
* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)
* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)
* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)
* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)
* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)
* Delete unnecessary javascript views. [#4059](https://github.com/diaspora/diaspora/pull/4059)
## Bug fixes
* reset comment box height after posting a comment. [#4030](https://github.com/diaspora/diaspora/issues/4030)
@ -13,15 +110,6 @@
* Fix mobile view of deleted reshares. [#4063](https://github.com/diaspora/diaspora/issues/4063)
* Hide comment button in the mobile view when not signed in. [#4065](https://github.com/diaspora/diaspora/issues/4065)
## Refactor
* Delete unnecessary javascript views. [#4059] (https://github.com/diaspora/diaspora/pull/4059)
* Add a configuration entry to set max-age header to Amazon S3 resources. [#4048](https://github.com/diaspora/diaspora/pull/4048)
* Refactor people_controller#show and photos_controller#index [#4002](https://github.com/diaspora/diaspora/issues/4002)
* Modularize layout [#3944](https://github.com/diaspora/diaspora/pull/3944)
* Add header to the sign up page [#3944](https://github.com/diaspora/diaspora/pull/3944)
* Load images via sprites [#4039](https://github.com/diaspora/diaspora/pull/4039)
## Features
* Deleting a post that was shared to Facebook now deletes it from Facebook too [#3980]( https://github.com/diaspora/diaspora/pull/3980)

5
Gemfile
View file

@ -17,8 +17,9 @@ gem 'devise', '2.1.3'
# Background processing
gem 'resque', '1.23.0'
gem 'resque-timeout', '1.0.0'
gem 'sidekiq', '2.7.5'
gem 'sinatra', '1.3.3'
gem 'slim', '1.3.6'
# Configuration

30
Gemfile.lock
View file

@ -62,6 +62,9 @@ GEM
carrierwave (0.8.0)
activemodel (>= 3.2.0)
activesupport (>= 3.2.0)
celluloid (0.12.4)
facter (>= 1.6.12)
timers (>= 1.0.0)
childprocess (0.3.9)
ffi (~> 1.0, >= 1.0.11)
chunky_png (1.2.7)
@ -81,6 +84,7 @@ GEM
compass-rails (1.0.3)
compass (>= 0.12.2, < 0.14)
configurate (0.0.2)
connection_pool (1.0.0)
crack (0.3.2)
cucumber (1.2.3)
builder (>= 2.1.2)
@ -103,6 +107,7 @@ GEM
excon (0.20.1)
execjs (1.4.0)
multi_json (~> 1.0)
facter (1.6.17)
factory_girl (4.2.0)
activesupport (>= 3.0.0)
factory_girl_rails (4.2.1)
@ -321,13 +326,6 @@ GEM
redis-namespace (1.2.1)
redis (~> 3.0.0)
remotipart (1.0.5)
resque (1.23.0)
multi_json (~> 1.0)
redis-namespace (~> 1.0)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
resque-timeout (1.0.0)
resque (~> 1.0)
rmagick (2.13.2)
roxml (3.1.6)
activesupport (>= 2.3.0)
@ -363,11 +361,20 @@ GEM
multi_json (~> 1.0)
rubyzip
websocket (~> 1.0.4)
sidekiq (2.7.5)
celluloid (~> 0.12.0)
connection_pool (~> 1.0)
multi_json (~> 1)
redis (~> 3)
redis-namespace
simple_oauth (0.2.0)
sinatra (1.3.3)
rack (~> 1.3, >= 1.3.6)
rack-protection (~> 1.2)
tilt (~> 1.3, >= 1.3.3)
slim (1.3.6)
temple (~> 0.5.5)
tilt (~> 1.3.3)
slop (3.4.4)
spork (1.0.0rc3)
sprockets (2.2.2)
@ -376,10 +383,12 @@ GEM
rack (~> 1.0)
tilt (~> 1.1, != 1.3.0)
subexec (0.2.2)
temple (0.5.5)
terminal-table (1.4.5)
thor (0.17.0)
tilt (1.3.6)
timecop (0.6.1)
timers (1.1.0)
treetop (1.4.12)
polyglot
polyglot (>= 0.3.1)
@ -397,8 +406,6 @@ GEM
kgio (~> 2.6)
rack
raindrops (~> 0.7)
vegas (0.1.11)
rack (>= 1.0.0)
warden (1.2.1)
rack (>= 1.0)
webmock (1.8.11)
@ -473,8 +480,6 @@ DEPENDENCIES
rb-inotify (= 0.9.0)
redcarpet (= 2.2.2)
remotipart (= 1.0.5)
resque (= 1.23.0)
resque-timeout (= 1.0.0)
rmagick (= 2.13.2)
roxml (= 3.1.6)
rspec-instafail (= 0.2.4)
@ -482,6 +487,9 @@ DEPENDENCIES
ruby-oembed (= 0.8.8)
sass-rails (= 3.2.6)
selenium-webdriver (= 2.31.0)
sidekiq (= 2.7.5)
sinatra (= 1.3.3)
slim (= 1.3.6)
spork (= 1.0.0rc3)
timecop (= 0.6.1)
twitter (= 4.6.2)

2
Procfile
View file

@ -1,2 +1,2 @@
web: bundle exec unicorn_rails -c config/unicorn.rb -p $PORT
worker: env QUEUE=* bundle exec rake resque:work
sidekiq: bundle exec sidekiq

1
Rakefile
View file

@ -7,7 +7,6 @@
require File.expand_path('../config/application', __FILE__)
require 'rake'
require 'resque/tasks'
# for rake 0.9.0
module Diaspora

4
app/controllers/publics_controller.rb
View file

@ -49,7 +49,7 @@ class PublicsController < ApplicationController
def receive_public
FEDERATION_LOGGER.info("recieved a public message")
Resque.enqueue(Jobs::ReceiveUnencryptedSalmon, CGI::unescape(params[:xml]))
Workers::ReceiveUnencryptedSalmon.perform_async(CGI::unescape(params[:xml]))
render :nothing => true, :status => :ok
end
@ -65,7 +65,7 @@ class PublicsController < ApplicationController
@user = person.owner
FEDERATION_LOGGER.info("recieved a private message for user:#{@user.id}")
Resque.enqueue(Jobs::ReceiveEncryptedSalmon, @user.id, CGI::unescape(params[:xml]))
Workers::ReceiveEncryptedSalmon.perform_async(@user.id, CGI::unescape(params[:xml]))
render :nothing => true, :status => 202
end

2
app/controllers/services_controller.rb
View file

@ -36,7 +36,7 @@ class ServicesController < ApplicationController
fetch_photo = current_user.profile[:image_url].blank?
current_user.update_profile(current_user.profile.from_omniauth_hash(user))
Resque.enqueue(Jobs::FetchProfilePhoto, current_user.id, service.id, user["image"]) if fetch_photo
Workers::FetchProfilePhoto.perform_async(current_user.id, service.id, user["image"]) if fetch_photo
flash[:notice] = I18n.t 'services.create.success'
else

2
app/models/account_deletion.rb
View file

@ -26,7 +26,7 @@ class AccountDeletion < ActiveRecord::Base
end
def queue_delete_account
Resque.enqueue(Jobs::DeleteAccount, self.id)
Workers::DeleteAccount.perform_async(self.id)
end
def perform!

25
app/models/jobs/base.rb
View file

@ -1,25 +0,0 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
class Base
Dir[Rails.root.join('app', 'models', 'jobs', 'mail', '*.rb')].each {|file| require file }
#TODO these should be subclassed real exceptions
DUMB_ERROR_MESSAGES = [
"Contact required unless request",
"Relayable object, but no parent object found" ]
def self.suppress_annoying_errors(&block)
begin
yield
rescue => e
Rails.logger.info("error in job: #{e.message}")
unless DUMB_ERROR_MESSAGES.include?(e.message)
raise e
end
end
end
end
end

16
app/models/jobs/gather_o_embed_data.rb
View file

@ -1,16 +0,0 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
#
module Jobs
class GatherOEmbedData < Base
@queue = :http_service
def self.perform(post_id, url)
post = Post.find(post_id)
post.o_embed_cache = OEmbedCache.find_or_create_by_url(url)
post.save
end
end
end

17
app/models/jobs/receive.rb
View file

@ -1,17 +0,0 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
class Receive < Base
@queue = :receive
def self.perform(user_id, xml, salmon_author_id)
user = User.find(user_id)
salmon_author = Person.find(salmon_author_id)
zord = Postzord::Receiver::Private.new(user, :person => salmon_author)
zord.parse_and_receive(xml)
end
end
end

21
app/models/jobs/receive_unencrypted_salmon.rb
View file

@ -1,21 +0,0 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
require Rails.root.join('lib', 'postzord', 'receiver', 'public')
module Jobs
class ReceiveUnencryptedSalmon < Base
@queue = :receive
def self.perform(xml)
begin
receiver = Postzord::Receiver::Public.new(xml)
receiver.perform!
rescue => e
FEDERATION_LOGGER.info(e.message)
raise e
end
end
end
end

2
app/models/notifications/also_commented.rb
View file

@ -1,6 +1,6 @@
class Notifications::AlsoCommented < Notification
def mail_job
Jobs::Mail::AlsoCommented
Workers::Mail::AlsoCommented
end
def popup_translation_key

2
app/models/notifications/comment_on_post.rb
View file

@ -1,6 +1,6 @@
class Notifications::CommentOnPost < Notification
def mail_job
Jobs::Mail::CommentOnPost
Workers::Mail::CommentOnPost
end
def popup_translation_key

2
app/models/notifications/liked.rb
View file

@ -1,6 +1,6 @@
class Notifications::Liked < Notification
def mail_job
Jobs::Mail::Liked
Workers::Mail::Liked
end
def popup_translation_key

2
app/models/notifications/mentioned.rb
View file

@ -1,6 +1,6 @@
class Notifications::Mentioned < Notification
def mail_job
Jobs::Mail::Mentioned
Workers::Mail::Mentioned
end
def popup_translation_key

2
app/models/notifications/private_message.rb
View file

@ -1,6 +1,6 @@
class Notifications::PrivateMessage < Notification
def mail_job
Jobs::Mail::PrivateMessage
Workers::Mail::PrivateMessage
end
def popup_translation_key
'notifications.private_message'

2
app/models/notifications/request_accepted.rb
View file

@ -1,6 +1,6 @@
class Notifications::RequestAccepted < Notification
def mail_job
Jobs::Mail::RequestAcceptance
Workers::Mail::RequestAcceptance
end
def popup_translation_key
'notifications.request_accepted'

3
app/models/notifications/reshared.rb
View file

@ -1,7 +1,6 @@
class Notifications::Reshared < Notification
def mail_job
Jobs::Mail::Reshared
#Jobs::Mail::Liked
Workers::Mail::Reshared
end
def popup_translation_key

2
app/models/notifications/started_sharing.rb
View file

@ -1,6 +1,6 @@
class Notifications::StartedSharing < Notification
def mail_job
Jobs::Mail::StartedSharing
Workers::Mail::StartedSharing
end
def popup_translation_key

2
app/models/photo.rb
View file

@ -127,7 +127,7 @@ class Photo < ActiveRecord::Base
end
def queue_processing_job
Resque.enqueue(Jobs::ProcessPhoto, self.id)
Workers::ProcessPhoto.perform_async(self.id)
end
def mutable?

2
app/models/status_message.rb
View file

@ -155,7 +155,7 @@ class StatusMessage < Post
end
def queue_gather_oembed_data
Resque.enqueue(Jobs::GatherOEmbedData, self.id, self.oembed_url)
Workers::GatherOEmbedData.perform_async(self.id, self.oembed_url)
end
def contains_oembed_url_in_text?

8
app/models/user.rb
View file

@ -164,7 +164,7 @@ class User < ActiveRecord::Base
def send_reset_password_instructions
generate_reset_password_token! if should_generate_reset_token?
Resque.enqueue(Jobs::ResetPassword, self.id)
Workers::ResetPassword.perform_async(self.id)
end
def update_user_preferences(pref_hash)
@ -299,15 +299,15 @@ class User < ActiveRecord::Base
######### Mailer #######################
def mail(job, *args)
pref = job.to_s.gsub('Jobs::Mail::', '').underscore
pref = job.to_s.gsub('Workers::Mail::', '').underscore
if(self.disable_mail == false && !self.user_preferences.exists?(:email_type => pref))
Resque.enqueue(job, *args)
job.perform_async(*args)
end
end
def mail_confirm_email
return false if unconfirmed_email.blank?
Resque.enqueue(Jobs::Mail::ConfirmEmail, id)
Workers::Mail::ConfirmEmail.perform_async(id)
true
end

3
app/views/admins/_admin_bar.haml
View file

@ -6,6 +6,5 @@
%li= link_to t('.weekly_user_stats'), weekly_user_stats_path
%li= link_to t('.pod_stats'), pod_stats_path
%li= link_to t('.correlations'), correlations_path
- if AppConfig.admins.inline_resque_web?
%li= link_to t('.resque_overview'), resque_web_path
%li= link_to t('.sidekiq_monitor'), sidekiq_path

23
app/workers/base.rb Normal file
View file

@ -0,0 +1,23 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Workers
class Base
include Sidekiq::Worker
sidekiq_options timeout: AppConfig.environment.sidekiq.timeout.to_i,
backtrace: ((bt = AppConfig.environment.sidekiq.backtrace.get) && bt.to_i),
retry: AppConfig.environment.sidekiq.retry.to_i
# In the long term we need to eliminate the cause of these
def suppress_annoying_errors(&block)
yield
rescue Diaspora::ContactRequiredUnlessRequest,
Diaspora::RelayableObjectWithoutParent => e
Rails.logger.info("error on receive: #{e.class}")
rescue ActiveRecord::RecordInvalid => e
Rails.logger.info("failed to save received object: #{e.record.errors.full_messages}")
raise e unless e.message.match(/already been taken/)
end
end
end

6
app/models/jobs/deferred_dispatch.rb → app/workers/deferred_dispatch.rb
View file

@ -2,11 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class DeferredDispatch < Base
@queue = :dispatch
sidekiq_options queue: :dispatch
def self.perform(user_id, object_class_name, object_id, opts)
def perform(user_id, object_class_name, object_id, opts)
user = User.find(user_id)
object = object_class_name.constantize.find(object_id)
opts = HashWithIndifferentAccess.new(opts)

7
app/models/jobs/delete_account.rb → app/workers/delete_account.rb
View file

@ -3,10 +3,11 @@
# the COPYRIGHT file.
module Jobs
module Workers
class DeleteAccount < Base
@queue = :delete_account
def self.perform(account_deletion_id)
sidekiq_options queue: :delete_account
def perform(account_deletion_id)
account_deletion = AccountDeletion.find(account_deletion_id)
account_deletion.perform!
end

6
app/models/jobs/delete_post_from_service.rb → app/workers/delete_post_from_service.rb
View file

@ -2,11 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
#
module Jobs
module Workers
class DeletePostFromService < Base
@queue = :http_service
sidekiq_options queue: :http_service
def self.perform(service_id, service_post_id)
def perform(service_id, service_post_id)
service = Service.find_by_id(service_id)
service.delete_post(service_post_id)
end

7
app/models/jobs/fetch_profile_photo.rb → app/workers/fetch_profile_photo.rb
View file

@ -3,10 +3,11 @@
# the COPYRIGHT file.
module Jobs
module Workers
class FetchProfilePhoto < Base
@queue = :photos
def self.perform(user_id, service_id, fallback_image_url = nil)
sidekiq_options queue: :photos
def perform(user_id, service_id, fallback_image_url = nil)
service = Service.find(service_id)
image_url = service.profile_photo_url

10
app/models/jobs/fetch_public_posts.rb → app/workers/fetch_public_posts.rb
View file

@ -2,14 +2,12 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class FetchPublicPosts < Base
@queue = :http_service
sidekiq_options queue: :http_service
def self.perform(diaspora_id)
require Rails.root.join('lib','diaspora','fetcher','public')
PublicFetcher.new.fetch!(diaspora_id)
def perform(diaspora_id)
Diaspora::Fetcher::Public.new.fetch!(diaspora_id)
end
end
end

8
app/models/jobs/fetch_webfinger.rb → app/workers/fetch_webfinger.rb
View file

@ -2,15 +2,15 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class FetchWebfinger < Base
@queue = :socket_webfinger
sidekiq_options queue: :socket_webfinger
def self.perform(account)
def perform(account)
person = Webfinger.new(account).fetch
# also, schedule to fetch a few public posts from that person
Resque.enqueue(Jobs::FetchPublicPosts, person.diaspora_handle) unless person.nil?
Workers::FetchPublicPosts.perform_async(person.diaspora_handle) unless person.nil?
end
end
end

22
app/workers/gather_o_embed_data.rb Normal file
View file

@ -0,0 +1,22 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
#
module Workers
class GatherOEmbedData < Base
sidekiq_options queue: :http_service
def perform(post_id, url, retry_count=1)
post = Post.find(post_id)
post.o_embed_cache = OEmbedCache.find_or_create_by_url(url)
post.save
rescue ActiveRecord::RecordNotFound
# User created a post and deleted it right afterwards before we
# we had a chance to run the job.
# On the other hand sometimes the job runs before the Post is
# fully persisted. So we just reduce the amount of retries.
GatherOEmbedData.perform_in(1.minute, post_id, url, retry_count+1) unless retry_count > 3
end
end
end

14
app/models/jobs/http_multi.rb → app/workers/http_multi.rb
View file

@ -2,29 +2,25 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
require 'uri'
require Rails.root.join('lib', 'hydra_wrapper')
module Jobs
module Workers
class HttpMulti < Base
@queue = :http
sidekiq_options queue: :http
MAX_RETRIES = 3
def self.perform(user_id, encoded_object_xml, person_ids, dispatcher_class_as_string, retry_count=0)
def perform(user_id, encoded_object_xml, person_ids, dispatcher_class_as_string, retry_count=0)
user = User.find(user_id)
people = Person.where(:id => person_ids)
dispatcher = dispatcher_class_as_string.constantize
hydra = ::HydraWrapper.new(user, people, encoded_object_xml, dispatcher)
hydra = HydraWrapper.new(user, people, encoded_object_xml, dispatcher)
hydra.enqueue_batch
hydra.run
unless hydra.failed_people.empty?
if retry_count < MAX_RETRIES
Resque.enqueue(Jobs::HttpMulti, user_id, encoded_object_xml, hydra.failed_people, dispatcher_class_as_string, retry_count + 1 )
Workers::HttpMulti.perform_in(1.hour, user_id, encoded_object_xml, hydra.failed_people, dispatcher_class_as_string, retry_count + 1)
else
Rails.logger.info("event=http_multi_abandon sender_id=#{user_id} failed_recipient_ids='[#{person_ids.join(', ')}] '")
end

7
app/models/jobs/mail/also_commented.rb → app/workers/mail/also_commented.rb
View file

@ -1,8 +1,9 @@
module Jobs
module Workers
module Mail
class AlsoCommented < Base
@queue = :mail
def self.perform(recipient_id, sender_id, comment_id)
sidekiq_options queue: :mail
def perform(recipient_id, sender_id, comment_id)
if email = Notifier.also_commented(recipient_id, sender_id, comment_id)
email.deliver
end

7
app/models/jobs/mail/comment_on_post.rb → app/workers/mail/comment_on_post.rb
View file

@ -1,8 +1,9 @@
module Jobs
module Workers
module Mail
class CommentOnPost < Base
@queue = :mail
def self.perform(recipient_id, sender_id, comment_id)
sidekiq_options queue: :mail
def perform(recipient_id, sender_id, comment_id)
Notifier.comment_on_post(recipient_id, sender_id, comment_id).deliver
end
end

7
app/models/jobs/mail/confirm_email.rb → app/workers/mail/confirm_email.rb
View file

@ -1,8 +1,9 @@
module Jobs
module Workers
module Mail
class ConfirmEmail < Base
@queue = :mail
def self.perform(user_id)
sidekiq_options queue: :mail
def perform(user_id)
Notifier.confirm_email(user_id).deliver
end
end

8
app/models/jobs/mail/invite_user_by_email.rb → app/workers/mail/invite_user_by_email.rb
View file

@ -2,12 +2,12 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
module Mail
class InviteUserByEmail < Base
@queue = :mail
def self.perform(invite_id)
sidekiq_options queue: :mail
def perform(invite_id)
invite = Invitation.find(invite_id)
I18n.with_locale(invite.language) do
invite.send!

7
app/models/jobs/mail/liked.rb → app/workers/mail/liked.rb
View file

@ -1,8 +1,9 @@
module Jobs
module Workers
module Mail
class Liked < Base
@queue = :mail
def self.perform(recipient_id, sender_id, like_id)
sidekiq_options queue: :mail
def perform(recipient_id, sender_id, like_id)
Notifier.liked(recipient_id, sender_id, like_id).deliver
end
end

7
app/models/jobs/mail/mentioned.rb → app/workers/mail/mentioned.rb
View file

@ -3,11 +3,12 @@
# the COPYRIGHT file.
module Jobs
module Workers
module Mail
class Mentioned < Base
@queue = :mail
def self.perform(recipient_id, actor_id, target_id)
sidekiq_options queue: :mail
def perform(recipient_id, actor_id, target_id)
Notifier.mentioned( recipient_id, actor_id, target_id).deliver
end
end

7
app/models/jobs/mail/private_message.rb → app/workers/mail/private_message.rb
View file

@ -3,11 +3,12 @@
# the COPYRIGHT file.
module Jobs
module Workers
module Mail
class PrivateMessage < Base
@queue = :mail
def self.perform(recipient_id, actor_id, target_id)
sidekiq_options queue: :mail
def perform(recipient_id, actor_id, target_id)
Notifier.private_message( recipient_id, actor_id, target_id).deliver
end
end

7
app/models/jobs/mail/reshared.rb → app/workers/mail/reshared.rb
View file

@ -1,8 +1,9 @@
module Jobs
module Workers
module Mail
class Reshared < Base
@queue = :mail
def self.perform(recipient_id, sender_id, reshare_id)
sidekiq_options queue: :mail
def perform(recipient_id, sender_id, reshare_id)
Notifier.reshared(recipient_id, sender_id, reshare_id).deliver
end
end

7
app/models/jobs/mail/started_sharing.rb → app/workers/mail/started_sharing.rb
View file

@ -3,11 +3,12 @@
# the COPYRIGHT file.
module Jobs
module Workers
module Mail
class StartedSharing < Base
@queue = :mail
def self.perform(recipient_id, sender_id, target_id)
sidekiq_options queue: :mail
def perform(recipient_id, sender_id, target_id)
Notifier.started_sharing(recipient_id, sender_id).deliver
end
end

8
app/models/jobs/notify_local_users.rb → app/workers/notify_local_users.rb
View file

@ -2,13 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class NotifyLocalUsers < Base
@queue = :receive_local
sidekiq_options queue: :receive_local
require Rails.root.join('app', 'models', 'notification')
def self.perform(user_ids, object_klass, object_id, person_id)
def perform(user_ids, object_klass, object_id, person_id)
object = object_klass.constantize.find_by_id(object_id)

6
app/models/jobs/post_to_service.rb → app/workers/post_to_service.rb
View file

@ -2,11 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
#
module Jobs
module Workers
class PostToService < Base
@queue = :http_service
sidekiq_options queue: :http_service
def self.perform(service_id, post_id, url)
def perform(service_id, post_id, url)
service = Service.find_by_id(service_id)
post = Post.find_by_id(post_id)
service.post(post, url)

7
app/models/jobs/process_photo.rb → app/workers/process_photo.rb
View file

@ -3,10 +3,11 @@
# the COPYRIGHT file.
module Jobs
module Workers
class ProcessPhoto < Base
@queue = :photos
def self.perform(id)
sidekiq_options queue: :photos
def perform(id)
photo = Photo.find(id)
unprocessed_image = photo.unprocessed_image

7
app/models/jobs/publish_to_hub.rb → app/workers/publish_to_hub.rb
View file

@ -2,12 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class PublishToHub < Base
@queue = :http_service
sidekiq_options queue: :http_service
def self.perform(sender_public_url)
require Rails.root.join('lib', 'pubsubhubbub')
def perform(sender_public_url)
atom_url = sender_public_url + '.atom'
Pubsubhubbub.new(AppConfig.environment.pubsub_server.get).publish(atom_url)
end

19
app/workers/receive.rb Normal file
View file

@ -0,0 +1,19 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Workers
class Receive < Base
sidekiq_options queue: :receive
def perform(user_id, xml, salmon_author_id)
suppress_annoying_errors do
user = User.find(user_id)
salmon_author = Person.find(salmon_author_id)
zord = Postzord::Receiver::Private.new(user, :person => salmon_author)
zord.parse_and_receive(xml)
end
end
end
end

7
app/models/jobs/receive_encrypted_salmon.rb → app/workers/receive_encrypted_salmon.rb
View file

@ -3,12 +3,11 @@
# the COPYRIGHT file.
require Rails.root.join('lib', 'postzord', 'receiver', 'private')
module Jobs
module Workers
class ReceiveEncryptedSalmon < Base
@queue = :receive_salmon
sidekiq_options queue: :receive_salmon
def self.perform(user_id, xml)
def perform(user_id, xml)
suppress_annoying_errors do
user = User.find(user_id)
zord = Postzord::Receiver::Private.new(user, :salmon_xml => xml)

10
app/models/jobs/receive_local_batch.rb → app/workers/receive_local_batch.rb
View file

@ -2,15 +2,11 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
require Rails.root.join('lib', 'postzord', 'receiver', 'private')
require Rails.root.join('lib', 'postzord', 'receiver', 'local_batch')
module Jobs
module Workers
class ReceiveLocalBatch < Base
sidekiq_options queue: :receive
@queue = :receive
def self.perform(object_class_string, object_id, recipient_user_ids)
def perform(object_class_string, object_id, recipient_user_ids)
object = object_class_string.constantize.find(object_id)
receiver = Postzord::Receiver::LocalBatch.new(object, recipient_user_ids)
receiver.perform!

21
app/workers/receive_unencrypted_salmon.rb Normal file
View file

@ -0,0 +1,21 @@
# Copyright (c) 2010-2011, Diaspora Inc. This file is
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Workers
class ReceiveUnencryptedSalmon < Base
sidekiq_options queue: :receive
def perform(xml)
suppress_annoying_errors do
begin
receiver = Postzord::Receiver::Public.new(xml)
receiver.perform!
rescue => e
FEDERATION_LOGGER.info(e.message)
raise e
end
end
end
end
end

7
app/models/jobs/resend_invitation.rb → app/workers/resend_invitation.rb
View file

@ -3,10 +3,11 @@
# the COPYRIGHT file.
module Jobs
module Workers
class ResendInvitation < Base
@queue = :mail
def self.perform(invitation_id)
sidekiq_options queue: :mail
def perform(invitation_id)
inv = Invitation.find(invitation_id)
inv.resend
end

6
app/models/jobs/reset_password.rb → app/workers/reset_password.rb
View file

@ -1,8 +1,8 @@
module Jobs
module Workers
class ResetPassword < Base
@queue = :mail
sidekiq_options queue: :mail
def self.perform(user_id)
def perform(user_id)
user = User.find(user_id)
::Devise.mailer.reset_password_instructions(user).deliver
end

19
config/database.yml.example
View file

@ -18,10 +18,13 @@ postgres: &postgres
# Comment the the mysql line and uncomment the postgres line
# if you want to use postgres
choose: &choose
common: &common
# Choose one of the following
<<: *mysql
#<<: *postgres
# Should match environment.sidekiq.concurrency
#pool: 25
##################################################
#### CONFIGURE ABOVE #############################
@ -32,20 +35,20 @@ choose: &choose
postgres_travis: &postgres_travis
adapter: postgresql
username: postgres
common: &common
<<: *choose
development:
combined: &combined
<<: *common
development:
<<: *combined
database: diaspora_development
production:
<<: *common
<<: *combined
database: diaspora_production
test:
<<: *common
<<: *combined
database: "diaspora_test"
integration1:
<<: *common
<<: *combined
database: diaspora_integration1
integration2:
<<: *common
<<: *combined
database: diaspora_integration2

13
config/defaults.yml
View file

@ -12,6 +12,13 @@ defaults:
redis:
require_ssl: true
single_process_mode: false
sidekiq:
namespace: "diaspora"
concurrency: 5
retry: 10
timeout: 900
backtrace: 15
log: "log/sidekiq.log"
s3:
enable: false
key:
@ -32,8 +39,7 @@ defaults:
stdout_log:
database: 'mysql'
unicorn_worker: 2
embed_resque_worker: false
resque_workers: 1
embed_sidekiq_worker: false
privacy:
jquery_cdn: true
google_analytics_key:
@ -88,7 +94,6 @@ defaults:
admins:
account:
podmin_email:
inline_resque_web: true
development:
environment:
@ -123,12 +128,14 @@ test:
integration1:
environment:
url: "http://localhost:45789/"
single_process_mode: true
assets:
serve: true
require_ssl: false
integration2:
environment:
url: "http://localhost:34658/"
single_process_mode: true
assets:
serve: true
require_ssl: false

44
config/diaspora.yml.example
View file

@ -66,6 +66,38 @@ configuration: ## Section
## all the time intensive jobs must be run inside the request cycle.
## So this is higly unrecommended for production setups.
#single_process_mode: true
## Sidekiq - background processing
sidekiq: ## Section
## Number of parallel threads Sidekiq uses
## If you touch this please set the pool setting
## in your database.yml to a value that's at minimum
## close to this! The default value is 5 but you can safely
## increase it to 25 and more on a medium sized pod.
## This applies per started Sidekiq worker, so if you set it to
## 25 and start two workers you'll process up to 50 jobs in parallel.
#concurrency: 25
## Number of times a job is retried
## There's an exponential backoff, if you set this too
## high you might get too many jobs in the queue
## Set this to false to disable it completely
#retry: 10
## Time in seconds before a job is killed
#timeout: 900
## Namespace to use in Redis, useful if you need to run
## multiple instances of Diaspora using the same Redis instance
#namespace: "diaspora"
## Lines of backtrace that is stored on failure
## Set this to false if you're not interested in this data to
## reduce memory usage (and log size)
#backtrace: 15
## Log file for Sidekiq
#log: "log/sidekiq.log"
## Use Amazon S3 instead of your local filesystem
## to handle uploaded pictures.
@ -124,12 +156,9 @@ configuration: ## Section
## you have many users
#unicorn_worker: 2
## Embed a resque worker inside the unicorn process, useful for
## Embed a Sidekiq worker inside the unicorn process, useful for
## minimal Heroku setups
#embed_resque_worker: true
## Number of resque workers to start
#resque_workers: 1
#embed_sidekiq_worker: true
## Settings probably affecting the privacy of your users
privacy: ## Section
@ -293,11 +322,6 @@ configuration: ## Section
## E-Mail address users can contact the administrator
#podmin_email: 'podmin@example.org'
## Resque is the background processing system used by Diaspora
## Resque web is an admin tool for it. This settings decides whether
## or not to inline it into Diaspora.
#inline_resque_web: true
## Here you can make overides to settings defined above if you need
## to have them different in different environments.

1
config/initializers/load_mail_jobs.rb
View file

@ -1 +0,0 @@
Dir[Rails.root.join('app', 'models', 'jobs', 'mail', '*.rb')].each { |file| require file }

36
config/initializers/resque.rb
View file

@ -1,36 +0,0 @@
require 'resque'
Resque::Plugins::Timeout.timeout = 300
if !AppConfig.environment.single_process_mode?
Resque.redis = AppConfig.get_redis_instance
end
# Single process-mode hooks using Resque.inline
if AppConfig.environment.single_process_mode?
if Rails.env == 'production'
puts "WARNING: You are running Diaspora in production without Resque"
puts " workers turned on. Please set single_process_mode to false in"
puts " config/diaspora.yml."
end
Resque.inline = true
end
if AppConfig.admins.monitoring.airbrake_api_key.present?
require 'resque/failure/multiple'
require 'resque/failure/airbrake'
require 'resque/failure/redis'
Resque::Failure::Airbrake.configure do |config|
config.api_key = AppConfig.admins.monitoring.airbrake_api_key
config.secure = true
end
Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
Resque::Failure.backend = Resque::Failure::Multiple
end
if AppConfig.admins.inline_resque_web?
require 'resque/server'
require Rails.root.join('lib', 'admin_rack')
Resque::Server.use AdminRack
end

50
config/initializers/sidekiq.rb Normal file
View file

@ -0,0 +1,50 @@
require 'sidekiq_middlewares'
# Single process-mode
if AppConfig.environment.single_process_mode? && Rails.env != "test"
if Rails.env == 'production'
puts "WARNING: You are running Diaspora in production without Sidekiq"
puts " workers turned on. Please set single_process_mode to false in"
puts " config/diaspora.yml."
end
require 'sidekiq/testing/inline'
end
Sidekiq.configure_server do |config|
config.redis = AppConfig.get_redis_options
config.options = config.options.merge({
concurrency: AppConfig.environment.sidekiq.concurrency.to_i,
queues: %w{
socket_webfinger
photos
http_service
dispatch
mail
delete_account
receive_local
receive
receive_salmon
http
default
}
})
config.server_middleware do |chain|
chain.add SidekiqMiddlewares::CleanAndShortBacktraces
end
Sidekiq::Logging.initialize_logger AppConfig.sidekiq_log unless AppConfig.heroku?
# Set connection pool on Heroku
database_url = ENV['DATABASE_URL']
if(database_url)
ENV['DATABASE_URL'] = "#{database_url}?pool=#{AppConfig.environment.sidekiq.concurrency.get}"
ActiveRecord::Base.establish_connection
end
end
Sidekiq.configure_client do |config|
config.redis = AppConfig.get_redis_options
end

2
config/locales/diaspora/en.yml
View file

@ -89,7 +89,7 @@ en:
weekly_user_stats: "Weekly User Stats"
pod_stats: "Pod Stats"
correlations: "Correlations"
resque_overview: "Resque Overview"
sidekiq_monitor: "Sidekiq monitor"
correlations:
correlations_count: "Correlations with Sign In Count:"
user_search:

11
config/routes.rb
View file

@ -2,11 +2,17 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
require 'sidekiq/web'
Diaspora::Application.routes.draw do
if Rails.env.production?
mount RailsAdmin::Engine => '/admin_panel', :as => 'rails_admin'
end
constraints ->(req) { req.env["warden"].authenticate?(scope: :user) &&
req.env['warden'].user.admin? } do
mount Sidekiq::Web => '/sidekiq', :as => 'sidekiq'
end
get "/atom.xml" => redirect('http://blog.diasporafoundation.org/feed/atom') #too many stupid redirects :()
@ -206,11 +212,6 @@ Diaspora::Application.routes.draw do
#Protocol Url
get 'protocol' => redirect("https://github.com/diaspora/diaspora/wiki/Diaspora%27s-federation-protocol")
# Resque web
if AppConfig.admins.inline_resque_web?
mount Resque::Server.new, :at => '/resque-jobs', :as => "resque_web"
end
# Startpage
root :to => 'home#show'
end

17
config/unicorn.rb
View file

@ -15,7 +15,7 @@ preload_app true
# How long to wait before killing an unresponsive worker
timeout 30
@resque_pid = nil
@sidekiq_pid = nil
#pid '/var/run/diaspora/diaspora.pid'
#listen '/var/run/diaspora/diaspora.sock', :backlog => 2048
@ -33,14 +33,12 @@ before_fork do |server, worker|
ActiveRecord::Base.connection.disconnect!
# disconnect redis if in use
if !AppConfig.single_process_mode?
Resque.redis.client.disconnect
unless AppConfig.single_process_mode?
Sidekiq.redis {|redis| redis.client.disconnect }
end
if AppConfig.server.embed_resque_worker?
# Clean up Resque workers killed by previous deploys/restarts
Resque.workers.each { |w| w.unregister_worker }
@resque_pid ||= spawn('bundle exec rake resque:work QUEUES=*')
if AppConfig.server.embed_sidekiq_worker?
@sidekiq_pid ||= spawn('bundle exec sidekiq')
end
old_pid = '/var/run/diaspora/diaspora.pid.oldbin'
@ -58,9 +56,8 @@ after_fork do |server, worker|
# If using preload_app, enable this line
ActiveRecord::Base.establish_connection
# copy pasta from resque.rb because i'm a bad person
if !AppConfig.environment.single_process_mode?
Resque.redis = AppConfig.get_redis_instance
unless AppConfig.environment.single_process_mode?
Sidekiq.redis = AppConfig.get_redis_options
end
# Enable this line to have the workers run as different user/group

2
db/seeds.rb
View file

@ -46,7 +46,7 @@ Role.add_admin(bob.person)
puts "done!"
require Rails.root.join('spec', 'support', 'fake_resque')
require 'sidekiq/testing/inline'
require Rails.root.join('spec', 'support', 'user_methods')
print "Seeding post data..."

6
features/support/env.rb
View file

@ -46,16 +46,16 @@ prefork = proc do
require File.join(File.dirname(__FILE__), "integration_sessions_controller")
require File.join(File.dirname(__FILE__), "poor_mans_webmock")
require 'sidekiq/testing/inline'
require Rails.root.join('spec', 'helper_methods')
require Rails.root.join('spec', 'support', 'inlined_jobs')
require Rails.root.join('spec', 'support', 'user_methods')
include HelperMethods
# require 'webmock/cucumber'
# WebMock.disable_net_connect!(:allow_localhost => true)
require Rails.root.join('spec', 'support', 'fake_resque')
require File.join(File.dirname(__FILE__), 'run_resque_in_process')
#hax to get rubymine to run spork, set RUBYMINE_HOME in your .bash_profile
if ENV["RUBYMINE_HOME"]

16
features/support/poor_mans_webmock.rb
View file

@ -2,32 +2,28 @@
# licensed under the Affero General Public License version 3 or later. See
# the COPYRIGHT file.
module Jobs
module Workers
class PublishToHub < Base
@queue = :http_service
def self.perform(sender_public_url)
def perform(sender_public_url)
# don't publish to pubsubhubbub in cucumber
end
end
class HttpMulti < Base
@queue = :http
def self.perform(user_id, enc_object_xml, person_ids, retry_count=0)
def perform(user_id, enc_object_xml, person_ids, retry_count=0)
# don't federate in cucumber
end
end
class HttpPost < Base
@queue = :http
def self.perform(url, body, tries_remaining = NUM_TRIES)
def perform(url, body, tries_remaining = NUM_TRIES)
# don't post to outside services in cucumber
end
end
class PostToService < Base
@queue = :http_service
def self.perform(service_id, post_id, url)
def perform(service_id, post_id, url)
# don't post to services in cucumber
end
end
end
end

5
features/support/run_resque_in_process.rb
View file

@ -1,5 +0,0 @@
module Resque
def enqueue(klass, *args)
klass.send(:perform, *args)
end
end

19
lib/admin_rack.rb
View file

@ -1,19 +0,0 @@
class AdminRack
def initialize(app)
@app = app
end
def call(env)
user = env['warden'].authenticate(:scope => :user)
if user && user.admin?
@app.call(env)
else
[307, {"Location" => '/'}, self]
end
end
def each(&block)
end
end

16
lib/configuration_methods.rb
View file

@ -74,7 +74,7 @@ module Configuration
(git_revision || version)[0..8]
end
def get_redis_instance
def get_redis_options
if redistogo_url.present?
$stderr.puts "WARNING: using the REDISTOGO_URL environment variable is deprecated, please use REDIS_URL now."
ENV['REDIS_URL'] = redistogo_url
@ -85,17 +85,25 @@ module Configuration
redis_url = ENV['REDIS_URL'] || environment.redis.get
if ENV['RAILS_ENV']== 'integration2'
redis_options = { :host => 'localhost', :port => 6380 }
redis_options[:url] = "redis://localhost:6380"
elsif redis_url.present?
unless redis_url.start_with?("redis://") || redis_url.start_with?("unix:///")
$stderr.puts "WARNING: Your redis url (#{redis_url}) doesn't start with redis:// or unix:///"
end
redis_options = { :url => redis_url }
redis_options[:url] = redis_url
end
Redis.new(redis_options.merge(:thread_safe => true))
redis_options[:namespace] = AppConfig.environment.sidekiq_namespace.get
redis_options
end
def sidekiq_log
path = Pathname.new environment.sidekiq.log.get
path = Rails.root.join(path) unless pathname.absolute?
path.to_s
end
private
def get_git_info

13
lib/diaspora/exceptions.rb
View file

@ -17,4 +17,17 @@ module Diaspora
class NotMine < StandardError
end
# Received a message without having a contact
class ContactRequiredUnlessRequest < StandardError
end
# Got a relayable (comment, like etc.) without having the parent
class RelayableObjectWithoutParent < StandardError
end
# After building an object the author doesn't match the one in the
# original XML message
class AuthorXMLAuthorMismatch < StandardError
end
end

2
lib/diaspora/fetcher/public.rb
View file

@ -75,7 +75,7 @@ module Diaspora; module Fetcher; class Public
FEDERATION_LOGGER.info "fetching public posts for #{@person.diaspora_handle}"
resp = Faraday.get("#{@person.url}/people/#{@person.guid}") do |req|
resp = Faraday.get("#{@person.url}people/#{@person.guid}") do |req|
req.headers[:accept] = 'application/json'
req.headers[:user_agent] = 'diaspora-fetcher'
end

2
lib/diaspora/parser.rb
View file

@ -13,7 +13,7 @@ module Diaspora
rescue NameError => e
# A pods is trying to federate an object we don't recognize.
# i.e. their codebase is different from ours. Quietly discard
# so that no Resque job failure is created
# so that no job failure is created
nil
end
end

27
lib/postzord/dispatcher.rb
View file

@ -40,7 +40,7 @@ class Postzord::Dispatcher
if opts[:additional_subscribers].present?
opts[:additional_subscribers] = [*opts[:additional_subscribers]].map(&:id)
end
Resque.enqueue(Jobs::DeferredDispatch, user.id, object.class.to_s, object.id, opts)
Workers::DeferredDispatch.perform_async(user.id, object.class.to_s, object.id, opts)
end
# @param object [Object]
@ -102,15 +102,16 @@ class Postzord::Dispatcher
queue_remote_delivery_job(remote_people)
end
# Enqueues a job in Resque
# Enqueues a job
# @param remote_people [Array<Person>] Recipients of the post on other pods
# @return [void]
def queue_remote_delivery_job(remote_people)
Resque.enqueue(Jobs::HttpMulti,
@sender.id,
Base64.strict_encode64(@object.to_diaspora_xml),
remote_people.map{|p| p.id},
self.class.to_s)
Workers::HttpMulti.perform_async(
@sender.id,
Base64.strict_encode64(@object.to_diaspora_xml),
remote_people.map{|p| p.id},
self.class.to_s
)
end
# @param people [Array<Person>] Recipients of the post
@ -121,7 +122,7 @@ class Postzord::Dispatcher
else
people.each do |person|
Rails.logger.info("event=push route=local sender=#{@sender.diaspora_handle} recipient=#{person.diaspora_handle} payload_type=#{@object.class}")
Resque.enqueue(Jobs::Receive, person.owner_id, @xml, @sender.person_id)
Workers::Receive.perform_async(person.owner_id, @xml, @sender.person_id)
end
end
end
@ -129,13 +130,13 @@ class Postzord::Dispatcher
# @param people [Array<Person>] Recipients of the post
def batch_deliver_to_local(people)
ids = people.map{ |p| p.owner_id }
Resque.enqueue(Jobs::ReceiveLocalBatch, @object.class.to_s, @object.id, ids)
Workers::ReceiveLocalBatch.perform_async(@object.class.to_s, @object.id, ids)
Rails.logger.info("event=push route=local sender=#{@sender.diaspora_handle} recipients=#{ids.join(',')} payload_type=#{@object.class}")
end
def deliver_to_hub
Rails.logger.debug("event=post_to_service type=pubsub sender_handle=#{@sender.diaspora_handle}")
Resque.enqueue(Jobs::PublishToHub, @sender.public_url)
Workers::PublishToHub.perform_async(@sender.public_url)
end
# @param url [String]
@ -146,12 +147,12 @@ class Postzord::Dispatcher
end
if @object.instance_of?(StatusMessage)
services.each do |service|
Resque.enqueue(Jobs::PostToService, service.id, @object.id, url)
Workers::PostToService.perform_async(service.id, @object.id, url)
end
end
if @object.instance_of?(SignedRetraction)
services.select { |service| service.respond_to? :delete_post }.each do |service|
Resque.enqueue(Jobs::DeletePostFromService, service.id, @object.target.facebook_id)
Workers::DeletePostFromService.perform_async(service.id, @object.target.facebook_id)
end
end
end
@ -168,7 +169,7 @@ class Postzord::Dispatcher
#temp hax
unless object_is_related_to_diaspora_hq?
Resque.enqueue(Jobs::NotifyLocalUsers, users.map{|u| u.id}, @object.class.to_s, @object.id, @object.author.id)
Workers::NotifyLocalUsers.perform_async(users.map{|u| u.id}, @object.class.to_s, @object.id, @object.author.id)
end
end

6
lib/postzord/receiver/private.rb
View file

@ -60,12 +60,12 @@ class Postzord::Receiver::Private < Postzord::Receiver
end
def validate_object
raise "Contact required unless request" if contact_required_unless_request
raise "Relayable object, but no parent object found" if relayable_without_parent?
raise Diaspora::ContactRequiredUnlessRequest if contact_required_unless_request
raise Diaspora::RelayableObjectWithoutParent if relayable_without_parent?
assign_sender_handle_if_request
raise "Author does not match XML author" if author_does_not_match_xml_author?
raise Diaspora::AuthorXMLAuthorMismatch if author_does_not_match_xml_author?
@object
end

4
lib/postzord/receiver/public.rb
View file

@ -36,7 +36,7 @@ class Postzord::Receiver::Public < Postzord::Receiver
@object.perform user if user
end
else
Resque.enqueue(Jobs::ReceiveLocalBatch, @object.class.to_s, @object.id, self.recipient_user_ids)
Workers::ReceiveLocalBatch.perform_async(@object.class.to_s, @object.id, self.recipient_user_ids)
true
end
end
@ -57,7 +57,7 @@ class Postzord::Receiver::Public < Postzord::Receiver
def save_object
@object = Diaspora::Parser::from_xml(@salmon.parsed_data)
raise "Object is not public" if object_can_be_public_and_it_is_not?
raise "Author does not match XML author" if author_does_not_match_xml_author?
raise Diaspora::AuthorXMLAuthorMismatch if author_does_not_match_xml_author?
@object.save! if @object && @object.respond_to?(:save!)
@object
end

13
lib/sidekiq_middlewares.rb Normal file
View file

@ -0,0 +1,13 @@
module SidekiqMiddlewares
class CleanAndShortBacktraces
def call(worker, item, queue)
yield
rescue Exception
backtrace = Rails.backtrace_cleaner.clean($!.backtrace)
backtrace.reject! { |line| line =~ /lib\/sidekiq_middlewares.rb/ }
limit = AppConfig.environment.sidekiq.backtrace.to_i
backtrace = [] if limit == 0
raise $!, $!.message, backtrace[0..limit]
end
end
end

18
lib/tasks/resque.rake
View file

@ -1,18 +0,0 @@
require 'resque/tasks'
task "resque:setup" do
require File.join(File.dirname(__FILE__), '..', '..', 'config', 'environment')
Rails.logger.info("event=resque_setup rails_env=#{Rails.env}")
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
desc 'clear your failure queue in resque. good for crons.'
task 'resque:clear_failed' => [:environment] do
puts "clearing resque failures"
Resque::Failure.clear
puts "complete!"
end

2
lib/webfinger.rb
View file

@ -19,7 +19,7 @@ class Webfinger
end
def self.in_background(account, opts={})
Resque.enqueue(Jobs::FetchWebfinger, account)
Workers::FetchWebfinger.perform_async(account)
end
#everything below should be private I guess

2
script/ci/build.sh
View file

@ -5,7 +5,7 @@
echo "Setting up database.yml for $DB"
cp config/database.yml.example config/database.yml
if [ "$DB" = "postgres" ]; then
sed -i 's/*choose/*postgres_travis/' config/database.yml
sed -i 's/*common/*postgres_travis/' config/database.yml
fi
command="bundle exec rake --trace ci:travis:${BUILD_TYPE}"

15
script/server
View file

@ -35,9 +35,8 @@ os=`uname -s`
eval $(bundle exec ruby ./script/get_config.rb \
port=server.port \
db=server.database \
workers=server.resque_workers \
single_process_mode=environment.single_process_mode?
embed_resque_worker=server.embed_resque_worker
embed_sidekiq_worker=server.embed_sidekiq_worker
)
if [ -z "$DB" ]; then
@ -45,10 +44,6 @@ if [ -z "$DB" ]; then
export DB
fi
if [ "$single_process_mode" = "true" -o "$embed_resque_worker" = "true" ]; then
workers=0
fi
args="$@"
prev_arg=''
for arg in $( echo $args | awk '{ for (i = 1; i <= NF; i++) print $i}')
@ -106,13 +101,11 @@ fi
# Start Diaspora
echo -n "Starting Diaspora in $RAILS_ENV mode on port $port "
if [ "$embed_resque_worker" = "true" ]; then
echo "with a resque worker embeded into unicorn."
if [ "$embed_sidekiq_worker" = "true" ]; then
echo "with a sidekiq worker embeded into unicorn."
elif [ "$single_process_mode" = "true" ]; then
echo "with job processing inside the request cycle."
else
echo "with $workers resque workers."
fi
echo ""
exec bundle exec foreman start -m "web=1,worker=$workers" -p $port
exec bundle exec foreman start -m "web=1,sidekiq=1" -p $port

9
spec/controllers/devise/passwords_controller_spec.rb
View file

@ -17,8 +17,9 @@ describe Devise::PasswordsController do
post :create, "user" => {"email" => "foo@example.com"}
response.should be_success
end
it "doesn't send email" do
Resque.should_not_receive(:enqueue)
Workers::ResetPassword.should_not_receive(:perform_async)
post :create, "user" => {"email" => "foo@example.com"}
end
end
@ -27,10 +28,10 @@ describe Devise::PasswordsController do
post :create, "user" => {"email" => alice.email}
response.should redirect_to(new_user_session_path)
end
it "sends email (enqueued to Resque)" do
Resque.should_receive(:enqueue).with(Jobs::ResetPassword, alice.id)
it "sends email (enqueued to Sidekiq)" do
Workers::ResetPassword.should_receive(:perform_async).with(alice.id)
post :create, "user" => {"email" => alice.email}
end
end
end
end
end

6
spec/controllers/publics_controller_spec.rb
View file

@ -33,7 +33,7 @@ describe PublicsController do
it 'enqueues a ReceiveUnencryptedSalmon job' do
xml = "stuff"
Resque.should_receive(:enqueue).with(Jobs::ReceiveUnencryptedSalmon, xml)
Workers::ReceiveUnencryptedSalmon.should_receive(:perform_async).with(xml)
post :receive_public, :xml => xml
end
end
@ -47,7 +47,7 @@ describe PublicsController do
end
it 'enqueues a receive job' do
Resque.should_receive(:enqueue).with(Jobs::ReceiveEncryptedSalmon, @user.id, xml).once
Workers::ReceiveEncryptedSalmon.should_receive(:perform_async).with(@user.id, xml).once
post :receive, "guid" => @user.person.guid.to_s, "xml" => xml
end
@ -60,7 +60,7 @@ describe PublicsController do
salmon_factory = Salmon::EncryptedSlap.create_by_user_and_activity(@user, xml2)
enc_xml = salmon_factory.xml_for(user2.person)
Resque.should_receive(:enqueue).with(Jobs::ReceiveEncryptedSalmon, @user.id, enc_xml).once
Workers::ReceiveEncryptedSalmon.should_receive(:perform_async).with(@user.id, enc_xml).once
post :receive, "guid" => @user.person.guid.to_s, "xml" => CGI::escape(enc_xml)
end

4
spec/controllers/services_controller_spec.rb
View file

@ -76,7 +76,7 @@ describe ServicesController do
profile[:image_url] = "/non/default/image.jpg"
profile.save
Resque.should_not_receive(:enqueue)
Workers::FetchProfilePhoto.should_not_receive(:perform_async)
post :create, :provider => 'twitter'
end
@ -86,7 +86,7 @@ describe ServicesController do
profile[:image_url] = nil
profile.save
Resque.should_receive(:enqueue).with(Jobs::FetchProfilePhoto, @user.id, anything(), "https://service.com/fallback_lowres.jpg")
Workers::FetchProfilePhoto.should_receive(:perform_async).with(@user.id, anything(), "https://service.com/fallback_lowres.jpg")
post :create, :provider => 'twitter'
end

7
spec/controllers/status_messages_controller_spec.rb
View file

@ -202,11 +202,12 @@ describe StatusMessagesController do
end
it "sets the pending bit of referenced photos" do
fantasy_resque do
inlined_jobs do
post :create, @hash
@photo1.reload.pending.should be_false
@photo2.reload.pending.should be_false
end
@photo1.reload.pending.should be_false
@photo2.reload.pending.should be_false
end
end
end

10
spec/controllers/users_controller_spec.rb
View file

@ -121,10 +121,6 @@ describe UsersController do
end
describe 'email' do
before do
Resque.stub!(:enqueue)
end
it 'disallow the user to change his new (unconfirmed) mail when it is the same as the old' do
@user.email = "my@newemail.com"
put(:update, :id => @user.id, :user => { :email => "my@newemail.com"})
@ -157,7 +153,7 @@ describe UsersController do
end
it 'sends out activation email on success' do
Resque.should_receive(:enqueue).with(Jobs::Mail::ConfirmEmail, @user.id).once
Workers::Mail::ConfirmEmail.should_receive(:perform_async).with(@user.id).once
put(:update, :id => @user.id, :user => { :email => "my@newemail.com"})
end
end
@ -209,7 +205,7 @@ describe UsersController do
describe '#destroy' do
it 'does nothing if the password does not match' do
Resque.should_not_receive(:enqueue)
Workers::DeleteAccount.should_not_receive(:perform_async)
delete :destroy, :user => { :current_password => "stuff" }
end
@ -219,7 +215,7 @@ describe UsersController do
end
it 'enqueues a delete job' do
Resque.should_receive(:enqueue).with(Jobs::DeleteAccount, anything)
Workers::DeleteAccount.should_receive(:perform_async).with(anything)
delete :destroy, :user => { :current_password => "bluepin7" }
end
end

26
spec/integration/attack_vectors_spec.rb
View file

@ -85,9 +85,9 @@ describe "attack vectors" do
zord = Postzord::Receiver::Private.new(bob, :salmon_xml => salmon_xml)
expect {
expect_error /Contact required/ do
expect {
zord.perform!
end
}.to raise_error Diaspora::ContactRequiredUnlessRequest
}.to_not change(Post, :count)
user_should_not_see_guid(bob, bad_post_guid)
@ -110,9 +110,9 @@ describe "attack vectors" do
#bob sends it to himself?????
zord = Postzord::Receiver::Private.new(bob, :salmon_xml => salmon_xml)
expect_error /Contact required/ do
expect {
zord.perform!
end
}.to raise_error Diaspora::ContactRequiredUnlessRequest
#alice still should not see eves original post, even though bob sent it to her
user_should_not_see_guid(alice, original_message.guid)
@ -125,9 +125,9 @@ describe "attack vectors" do
profile.first_name = "Not BOB"
expect {
expect_error /Author does not match XML author/ do
expect {
receive(profile, :from => alice, :by => bob)
end
}.to raise_error Diaspora::AuthorXMLAuthorMismatch
}.to_not change(eve.profile, :first_name)
end
end
@ -135,9 +135,9 @@ describe "attack vectors" do
it 'public stuff should not be spoofed from another author' do
post = FactoryGirl.build(:status_message, :public => true, :author => eve.person)
expect_error /Author does not match XML author/ do
expect {
receive_public(post, :from => alice)
end
}.to raise_error Diaspora::AuthorXMLAuthorMismatch
end
end
@ -209,9 +209,9 @@ describe "attack vectors" do
end
expect {
expect_error /Author does not match XML author/ do
expect {
receive(retraction, :from => alice, :by => bob)
end
}.to raise_error Diaspora::AuthorXMLAuthorMismatch
}.to_not change(bob.visible_shareables(Post), :count)
end
@ -239,10 +239,10 @@ describe "attack vectors" do
end
expect{
expect_error /Author does not match XML author/ do
expect {
receive(retraction, :from => alice, :by => bob)
end
}.to_not change(bob.contacts, :count)
}.to raise_error Diaspora::AuthorXMLAuthorMismatch
}.to_not change(bob.contacts, :count)
end
it 'does not let another user update other persons post' do

7
spec/integration/dispatching_spec.rb
View file

@ -7,13 +7,14 @@ describe "Dispatching" do
# Luke has a public post and comments on it
post = FactoryGirl.create(:status_message, :public => true, :author => luke.person)
fantasy_resque do
comment = luke.comment!(post, "awesomesauseum")
comment = luke.comment!(post, "awesomesauseum")
inlined_jobs do
# Luke now retracts his comment
Postzord::Dispatcher::Public.should_not_receive(:new)
Postzord::Dispatcher::Private.should_receive(:new).and_return(stub(:post => true))
luke.retract(comment)
end
end
end
end
end

15
spec/integration/receiving_spec.rb
View file

@ -42,11 +42,12 @@ describe 'a user receives a post' do
end
it "should show bob's post to alice" do
fantasy_resque do
inlined_jobs do |queue|
sm = bob.build_post(:status_message, :text => "hi")
sm.save!
bob.aspects.reload
bob.add_to_streams(sm, [@bobs_aspect])
queue.drain_all
bob.dispatch_post(sm, :to => @bobs_aspect)
end
@ -173,7 +174,7 @@ describe 'a user receives a post' do
context 'remote' do
before do
fantasy_resque do
inlined_jobs do |queue|
connect_users(alice, @alices_aspect, eve, @eves_aspect)
@post = alice.post(:status_message, :text => "hello", :to => @alices_aspect.id)
@ -183,6 +184,7 @@ describe 'a user receives a post' do
receive_with_zord(eve, alice.person, xml)
comment = eve.comment!(@post, 'tada')
queue.drain_all
# After Eve creates her comment, it gets sent to Alice, who signs it with her private key
# before relaying it out to the contacts on the top-level post
comment.parent_author_signature = comment.sign_with_key(alice.encryption_key)
@ -190,6 +192,7 @@ describe 'a user receives a post' do
comment.delete
comment_with_whitespace = alice.comment!(@post, ' I cannot lift my thumb from the spacebar ')
queue.drain_all
@xml_with_whitespace = comment_with_whitespace.to_diaspora_xml
@guid_with_whitespace = comment_with_whitespace.guid
comment_with_whitespace.delete
@ -253,13 +256,13 @@ describe 'a user receives a post' do
end
it 'does not raise a `Mysql2::Error: Duplicate entry...` exception on save' do
fantasy_resque do
inlined_jobs do
@comment = bob.comment!(@post, 'tada')
@xml = @comment.to_diaspora_xml
lambda {
expect {
receive_with_zord(alice, bob.person, @xml)
}.should_not raise_exception
}.to_not raise_exception
end
end
end

41
spec/lib/configuration_methods_spec.rb
View file

@ -95,14 +95,14 @@ describe Configuration::Methods do
end
end
describe "#get_redis_instance" do
describe "#get_redis_options" do
context "with REDISTOGO_URL set" do
before do
ENV["REDISTOGO_URL"] = "redis://myserver"
end
it "uses that" do
@settings.get_redis_instance.client.host.should == "myserver"
@settings.get_redis_options[:url].should match "myserver"
end
end
@ -113,7 +113,7 @@ describe Configuration::Methods do
end
it "uses that" do
@settings.get_redis_instance.client.host.should == "yourserver"
@settings.get_redis_options[:url].should match "yourserver"
end
end
@ -125,19 +125,7 @@ describe Configuration::Methods do
end
it "uses that" do
@settings.get_redis_instance.client.host.should == "ourserver"
end
end
context "with nothing set" do
before do
@settings.environment.redis = nil
ENV["REDISTOGO_URL"] = nil
ENV["REDIS_URL"] = nil
end
it "uses localhost" do
@settings.get_redis_instance.client.host.should == "127.0.0.1"
@settings.get_redis_options[:url].should match "ourserver"
end
end
@ -149,7 +137,26 @@ describe Configuration::Methods do
end
it "uses that" do
@settings.get_redis_instance.client.path.should == "/tmp/redis.sock"
@settings.get_redis_options[:url].should match "/tmp/redis.sock"
end
end
end
describe "sidekiq_log" do
context "with a relative log set" do
it "joins that with Rails.root" do
path = "/some/path/"
Rails.stub!(:root).and_return(stub(join: path))
@settings.environment.sidekiq.log = "relative_path"
@settings.sidekiq_log.should match path
end
end
context "with a absolute path" do
it "just returns that" do
path = "/foobar.log"
@settings.environment.sidekiq.log = path
@settings.sidekiq_log.should == path
end
end
end

28
spec/lib/postzord/dispatcher_spec.rb
View file

@ -209,7 +209,7 @@ describe Postzord::Dispatcher do
it 'should queue an HttpMultiJob for the remote people' do
Postzord::Dispatcher::Public.any_instance.unstub(:deliver_to_remote)
Resque.should_receive(:enqueue).with(Jobs::HttpMulti, alice.id, anything, @remote_people.map{|p| p.id}, anything).once
Workers::HttpMulti.should_receive(:perform_async).with(alice.id, anything, @remote_people.map{|p| p.id}, anything).once
@mailman.send(:deliver_to_remote, @remote_people)
Postzord::Dispatcher::Public.stub(:deliver_to_remote)
@ -224,18 +224,18 @@ describe Postzord::Dispatcher do
it 'queues a batch receive' do
local_people = []
local_people << alice.person
Resque.should_receive(:enqueue).with(Jobs::ReceiveLocalBatch, @sm.class.to_s, @sm.id, [alice.id]).once
Workers::ReceiveLocalBatch.should_receive(:perform_async).with(@sm.class.to_s, @sm.id, [alice.id]).once
@mailman.send(:deliver_to_local, local_people)
end
it 'returns if people are empty' do
Resque.should_not_receive(:enqueue)
Workers::ReceiveLocalBatch.should_not_receive(:perform_async)
@mailman.send(:deliver_to_local, [])
end
it 'returns if the object is a profile' do
@mailman.instance_variable_set(:@object, Profile.new)
Resque.should_not_receive(:enqueue)
Workers::ReceiveLocalBatch.should_not_receive(:perform_async)
@mailman.send(:deliver_to_local, [1])
end
end
@ -277,8 +277,8 @@ describe Postzord::Dispatcher do
end
it 'queues a job to notify the hub' do
Resque.stub!(:enqueue).with(Jobs::PostToService, anything, anything, anything)
Resque.should_receive(:enqueue).with(Jobs::PublishToHub, alice.public_url)
Workers::PostToService.stub!(:perform_async).with(anything, anything, anything)
Workers::PublishToHub.should_receive(:perform_async).with(alice.public_url)
@zord.send(:deliver_to_services, nil, [])
end
@ -297,17 +297,17 @@ describe Postzord::Dispatcher do
alice.services << @s2
mailman = Postzord::Dispatcher.build(alice, FactoryGirl.create(:status_message), :url => "http://joindiaspora.com/p/123", :services => [@s1])
Resque.stub!(:enqueue).with(Jobs::PublishToHub, anything)
Resque.stub!(:enqueue).with(Jobs::HttpMulti, anything, anything, anything)
Resque.should_receive(:enqueue).with(Jobs::PostToService, @s1.id, anything, anything)
Workers::PublishToHub.stub!(:perform_async).with(anything)
Workers::HttpMulti.stub!(:perform_async).with(anything, anything, anything)
Workers::PostToService.should_receive(:perform_async).with(@s1.id, anything, anything)
mailman.post
end
it 'does not push to services if none are specified' do
mailman = Postzord::Dispatcher.build(alice, FactoryGirl.create(:status_message), :url => "http://joindiaspora.com/p/123")
Resque.stub!(:enqueue).with(Jobs::PublishToHub, anything)
Resque.should_not_receive(:enqueue).with(Jobs::PostToService, anything, anything, anything)
Workers::PublishToHub.stub!(:perform_async).with(anything)
Workers::PostToService.should_not_receive(:perform_async).with(anything, anything, anything)
mailman.post
end
@ -315,7 +315,7 @@ describe Postzord::Dispatcher do
retraction = SignedRetraction.build(alice, FactoryGirl.create(:status_message))
mailman = Postzord::Dispatcher.build(alice, retraction, :url => "http://joindiaspora.com/p/123", :services => [@service])
Resque.should_receive(:enqueue).with(Jobs::DeletePostFromService, anything, anything)
Workers::DeletePostFromService.should_receive(:perform_async).with(anything, anything)
mailman.post
end
@ -324,7 +324,7 @@ describe Postzord::Dispatcher do
service = Services::Twitter.new(access_token: "nope")
mailman = Postzord::Dispatcher.build(alice, retraction, :url => "http://joindiaspora.com/p/123", :services => [service])
Resque.should_not_receive(:enqueue).with(Jobs::DeletePostFromService, anything, anything)
Workers::DeletePostFromService.should_not_receive(:perform_async).with(anything, anything)
mailman.post
end
end
@ -338,7 +338,7 @@ describe Postzord::Dispatcher do
describe '#notify_users' do
it 'enqueues a NotifyLocalUsers job' do
Resque.should_receive(:enqueue).with(Jobs::NotifyLocalUsers, [bob.id], @zord.object.class.to_s, @zord.object.id, @zord.object.author.id)
Workers::NotifyLocalUsers.should_receive(:perform_async).with([bob.id], @zord.object.class.to_s, @zord.object.id, @zord.object.author.id)
@zord.send(:notify_users, [bob])
end
end

6
spec/lib/postzord/receiver/public_spec.rb
View file

@ -61,13 +61,13 @@ describe Postzord::Receiver::Public do
@receiver.perform!
end
it 'enqueues a Jobs::ReceiveLocalBatch' do
Resque.should_receive(:enqueue).with(Jobs::ReceiveLocalBatch, anything, anything, anything)
it 'enqueues a Workers::ReceiveLocalBatch' do
Workers::ReceiveLocalBatch.should_receive(:perform_async).with(anything, anything, anything)
@receiver.perform!
end
it 'intergrates' do
fantasy_resque do
inlined_jobs do
@receiver.perform!
end
end

4
spec/lib/webfinger_spec.rb
View file

@ -31,8 +31,8 @@ describe Webfinger do
end
describe '.in_background' do
it 'enqueues a Jobs::FetchWebfinger job' do
Resque.should_receive(:enqueue).with(Jobs::FetchWebfinger, account)
it 'enqueues a Workers::FetchWebfinger job' do
Workers::FetchWebfinger.should_receive(:perform_async).with(account)
Webfinger.in_background(account)
end
end

4
spec/models/account_deletion_spec.rb
View file

@ -10,8 +10,8 @@ describe AccountDeletion do
a.diaspora_handle.should == alice.person.diaspora_handle
end
it 'fires a resque job after creation'do
Resque.should_receive(:enqueue).with(Jobs::DeleteAccount, anything)
it 'fires a job after creation'do
Workers::DeleteAccount.should_receive(:perform_async).with(anything)
AccountDeletion.create(:person => alice.person)
end

6
spec/models/photo_spec.rb
View file

@ -197,7 +197,7 @@ describe Photo do
describe 'remote photos' do
before do
Jobs::ProcessPhoto.perform(@saved_photo.id)
Workers::ProcessPhoto.new.perform(@saved_photo.id)
end
it 'should set the remote_photo on marshalling' do
@ -229,8 +229,8 @@ describe Photo do
end
describe '#queue_processing_job' do
it 'should queue a resque job to process the images' do
Resque.should_receive(:enqueue).with(Jobs::ProcessPhoto, @photo.id)
it 'should queue a job to process the images' do
Workers::ProcessPhoto.should_receive(:perform_async).with(@photo.id)
@photo.queue_processing_job
end
end

2
spec/models/status_message_spec.rb
View file

@ -339,7 +339,7 @@ STR
it 'should queue a GatherOembedData if it includes a link' do
sm = FactoryGirl.build(:status_message, :text => @message_text)
Resque.should_receive(:enqueue).with(Jobs::GatherOEmbedData, instance_of(Fixnum), instance_of(String))
Workers::GatherOEmbedData.should_receive(:perform_async).with(instance_of(Fixnum), instance_of(String))
sm.save
end

20
spec/models/user_spec.rb
View file

@ -587,22 +587,22 @@ describe User do
alice.disable_mail = false
alice.save
Resque.should_receive(:enqueue).with(Jobs::Mail::StartedSharing, alice.id, 'contactrequestid').once
alice.mail(Jobs::Mail::StartedSharing, alice.id, 'contactrequestid')
Workers::Mail::StartedSharing.should_receive(:perform_async).with(alice.id, 'contactrequestid').once
alice.mail(Workers::Mail::StartedSharing, alice.id, 'contactrequestid')
end
it 'does not enqueue a mail job if the correct corresponding job has a prefrence entry' do
it 'does not enqueue a mail job if the correct corresponding job has a preference entry' do
alice.user_preferences.create(:email_type => 'started_sharing')
Resque.should_not_receive(:enqueue)
alice.mail(Jobs::Mail::StartedSharing, alice.id, 'contactrequestid')
Workers::Mail::StartedSharing.should_not_receive(:perform_async)
alice.mail(Workers::Mail::StartedSharing, alice.id, 'contactrequestid')
end
it 'does not send a mail if disable_mail is set to true' do
alice.disable_mail = true
alice.save
alice.reload
Resque.should_not_receive(:enqueue)
alice.mail(Jobs::Mail::StartedSharing, alice.id, 'contactrequestid')
Workers::Mail::StartedSharing.should_not_receive(:perform_async)
alice.mail(Workers::Mail::StartedSharing, alice.id, 'contactrequestid')
end
end
@ -721,12 +721,12 @@ describe User do
describe '#mail_confirm_email' do
it 'enqueues a mail job on user with unconfirmed email' do
user.update_attribute(:unconfirmed_email, "alice@newmail.com")
Resque.should_receive(:enqueue).with(Jobs::Mail::ConfirmEmail, alice.id).once
Workers::Mail::ConfirmEmail.should_receive(:perform_async).with(alice.id).once
alice.mail_confirm_email.should eql(true)
end
it 'enqueues NO mail job on user without unconfirmed email' do
Resque.should_not_receive(:enqueue).with(Jobs::Mail::ConfirmEmail, alice.id)
Workers::Mail::ConfirmEmail.should_not_receive(:perform_async).with(alice.id)
alice.mail_confirm_email.should eql(false)
end
end
@ -843,7 +843,7 @@ describe User do
it "queues up a job to send the reset password instructions" do
user = FactoryGirl.create :user
Resque.should_receive(:enqueue).with(Jobs::ResetPassword, user.id)
Workers::ResetPassword.should_receive(:perform_async).with(user.id)
user.send_reset_password_instructions
end
end

1
spec/spec_helper.rb
View file

@ -19,6 +19,7 @@ prefork = proc do
require 'rspec/rails'
require 'webmock/rspec'
require 'factory_girl'
require 'sidekiq/testing'
include HelperMethods

19
spec/support/fake_resque.rb
View file

@ -1,19 +0,0 @@
module Resque
def enqueue(klass, *args)
if $process_queue
klass.send(:perform, *args)
else
true
end
end
end
module HelperMethods
def fantasy_resque
former_value = $process_queue
$process_queue = true
result = yield
$process_queue = former_value
result
end
end

8
spec/support/fake_typhoeus.rb
View file

@ -10,13 +10,13 @@ end
def disable_typhoeus
silence_warnings do
Jobs::HttpMulti.const_set('Hydra', FakeHydra)
Jobs::HttpMulti.const_set('Request', FakeHydraRequest)
Workers::HttpMulti.const_set('Hydra', FakeHydra)
Workers::HttpMulti.const_set('Request', FakeHydraRequest)
end
end
def enable_typhoeus
silence_warnings do
Jobs::HttpMulti.const_set('Hydra', Typhoeus::Hydra)
Jobs::HttpMulti.const_set('Request', Typhoeus::Request)
Workers::HttpMulti.const_set('Hydra', Typhoeus::Hydra)
Workers::HttpMulti.const_set('Request', Typhoeus::Request)
end
end

10
spec/support/inlined_jobs.rb Normal file
View file

@ -0,0 +1,10 @@
module HelperMethods
def inlined_jobs
Sidekiq::Worker.clear_all
result = yield Sidekiq::Worker
Sidekiq::Worker.drain_all
result
rescue NoMethodError
yield Sidekiq::Worker if block_given? # Never error out on our own
end
end

4
spec/support/user_methods.rb
View file

@ -7,13 +7,13 @@ class User
alias_method :share_with_original, :share_with
def share_with(*args)
fantasy_resque do
inlined_jobs do
share_with_original(*args)
end
end
def post(class_name, opts = {})
fantasy_resque do
inlined_jobs do
p = build_post(class_name, opts)
if p.save!
self.aspects.reload

Some files were not shown because too many files have changed in this diff Show more