I was the lead for Parse Push. I reverse engineered Resque on Go so we could let any async stage be switched to Go cleanly. I added a few features like multiple workers per Redis connection and soft acks that let the next job be pulled by the framework but still kept the server alive during a soft shutdown until the job completed.

After moving the APNs (v2) server to Go I was able to map the wonky networking to Resque much better. My show & tell that week was literally "this MacBook is now running higher throughout than our cluster of >100 push servers". For APNs, the rewrite was a boon of over 500x per server.

As a great fringe benefit, fewer Resque servers meant less Redis polling. CPU dropped from 97% to <10 IIRC, which helped save us the next Black Friday where Redis falling over from Push load often caused problems.

I don't recall how often data loss was a problem. One of the points that hadn't gone into a lot of detail was when certain ops tricks happened. For example, I recall that reading from secondaries was reserved for larger push customers only. These queries could be wildly inefficient, have results in the 10s of millions, and due to mongo bugs would lock up an entire database if they had a geo component in the query. In later versions of Mongo where the bugs were fixed and after another generation of the auto indexer we were able to send all traffic back to primaries.

Oh. And our ResqueGo outperformed Ruby so badly that we couldn't do a reasonable X% server transition from Ruby to Go by rebalancing the number of workers. It had to be done my making a new Go queue and doing % rollout of which queue work was sent to. We learned that the hard way once (though luckily there was no regression IIRC).